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I. Executive Summary 



The purpose of the study was to examine the impacts of the Maryland State Performance Assessment 
Program (MSPAP) and the Maryland Learning Outcomes (MLO’s) on school curriculum, classroom 
instruction and assessment practices, professional development, and student learning. The data sources 
included questio nnair es for principals, mathematics teachers, and students, and mathematics classroom 
instruction, assessment, and test preparation activities. Ninety elementary and middle schools in 
Maryland participated in the study during the 1996-97 year. Mathematics teachers in both the on-grades 
( 3 rd^ 5 th and 8 th) and off-grades ( 2 °d, and 7*) participated in the study. Students in two of the on- 
grades ( 5 th and 8 ^) and two of the off-grades (4th and 7 th) responded to the student questionnaires. 

Descriptive Results from the Questionnaires . Most of the principals (approximately 98%) and 
teachers (approximately 90%) indicated that they were moderately or very familiar with the purpose, 
content and format of MSPAP. The majority of the principals (93%) and teachers ( 66 %) also indicated 
that they were moderately or very familiar with the MLO’s. Further, the majority of the students 
indicated that they were famil iar with the purpose of MSPAP (65%) and how the student responses to 
MSPAP are evaluated (57%). With respect to supporting MSPAP, the majority of the principals 
(approximately 86 %) and teachers (approximately 68 %) indicated that they somewhat or strongly 
supported MSPAP and that their support has increased over the years. The majority of the principals 
( 79 %) were supportive of holding schools accountable for meeting the performance standards on 
MSPAP, whereas, less than half of the teachers (37%) were supportive of using MSPAP for such 
purposes. However, the majority of both the principals (94%) and teachers (77%) indicated that MSPAP 
was a useful tool for making positive changes in instruction. 

The maj ority of the principals and teachers indicated that both mathematical procedural knowledge 
and problem solving were assessed by MSPAP to a moderate or great extent and, in general, the majority 
of principals and teachers indicated that the emphasis was appropriate. However, more teachers 
indicated that the emphasis on problem solving (62%), as compared to procedural knowledge (42%), was 
appropriate. In addition, less than half (37%) of the on-grade (3^, 5^, and 8 ^) teachers somewhat or 
strongly agreed with the statement that MSPAP tasks that assess mathematics are developmentally 
appropriate for students. 

The majority of teachers indicated that MSPAP had a moderate or great impact on their classroom 
activities (82%) and that MSPAP’s impact increased over the years (83%). Further, the majority of the 
teachers indicated that they place a moderate or great emphasis on the Maryland Learning Outcomes in 
their classroom activities (72%) and the emphasis has increased over the years (56%). Most of the 
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teachers (89%) indicated that they made a general improvement in their daily instruction to help students 
prepare for MSPAP. More importantly, teachers were in agreement more with the statement that they 
improved daily instruction than with any of the other test preparation activities. In addition, the majority 
of the 5 th and 8 *h grade students (71%) indicated that the use of MSPAP-like tasks in their instruction 

helped them answer the MSPAP tasks that assess mathematics to a moderate or great extent. 

« 

With respect to professional development activities, the majority of die teachers indicated that 
MSPAP (approximately 75%) and the MLO’s (56%) were addressed to a moderate or great extent in 
professional development activities. Further, the majority of the teachers indicated that they had a 
moderate or great amount of support and/or resources to enable them to make instruction (63%) and 
assessment (56%) changes to better reflect what is expected of students by MSPAP and the MLO’s. 

Statistica l Results from die Questionnaires . Confirmatory factor analyses supported six dimensions 
underlying the teacher questionnaire: MSPAP familiarity. Support of MSPAP, Current Math Instruction 
(extent to which instruction reflects the learning outcomes and reform-oriented problem types), Change 
in Math Instruction (extent to which teachers have changed their instruction to focus on the learning 
outcomes and reform-oriented problem types), MSPAP’s Impact on Instruction, and Professional 
Development Support related to MSPAP. Overall, teacher mean scores were at the upper end (favorable) 
of die score scale for each of these dimension. There were significant differences between grades for 
each of these dimensions except for the dimensions. Support MSPAP. In general, elementary on-grade 
teachers (3*^ and 5 th grades) were more favorable than middle school on- and off-grade teachers ( 7 th 
and 8 th) f or these dimensions. However, the differences were relatively small. 

Elementary and middle school principals responded similarly to the four dimensions underlying their 
questionnaires: MSPAP familiarity, Support MSPAP, MSPAP Impact, and Professional Development 
Support Further, both elementary and middle school principals, as compared to elementary and middle 
school teachers of mathematics, indicated that they were more familiar with MSPAP, that they were 
more supportive of MSPAP, that MSPAP had a greater impact on classroom instruction, and that 
teachers were receiving more professional development support related to MSPAP. 

Students in 4^, 5*h, 7 th, and 8 th grade responded to some of the same questions as teachers 
regarding their current math instruction. Fifth grade students (classes), as compared to 7 th and 8 th grade 
classes, indicated that their instruction was more reflective of fee learning outcomes and reform-oriented 
problem types. Further, 4 th, 5 th, 7 th and 8 th grade teachers, as compared to their students, were more 
likely to indicate feat their mathematics instruction had a grater emphasis on fee learning outcomes and 
reform oriented problem types. 
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Descriptive Results from the Classroom Activities . The results from the analyses of the classroom 
activities indicate, to some extent, that teachers’ assessment and instruction activities reflect the MLO’s, 
and have some of the same characteristics as the MSPAP tasks. Approximately half of the classroom 
activities, to some extent, required problem solving and reasoning, were integrated with another subject 
area (primarily writing), and required the application of skills and knowledge to real life problems. Over 
50% of the classroom activities required knowledge of numbers and operations, approximately 30% 
required geometry and/or measurement, and approximately 20% required statistics. The content outcome 
of probability, patterns and relationships, prealgebra and algebra were represented in less than 10% of 
the activities. However, algebra was represented more in 7^* and 8^* grade activities (17%-39%). 

Over one-third of die activities required students to provide at least one written explanation, while 
approximately 15% required several explanations and were very similar in form to the MSPAP tasks. 
Slightly less than one-third of the assessment activities included multiple-choice items. As might be 
expected the MSPAP test preparation activities that teachers used were more similar to die MSPAP tasks 
than were the classroom instruction and assessment activities. 

When differences across grade levels occurred with respect to the mathematics classroom activities, 
in most cases, they were in favor of the elementary school on-grades (3*^ and 5*^). As an example, a 
larger percentage of the 3”* and 5*h grade mathematics classroom instruction activities required students 
to problem solve and/or reason as compared to the other grade levels (2 Q d, 4^, 7^* and 8^). 

MSPAP Performance Gains From 1993-97 . The analyses of performance over time indicated, for 
the schools participating in this study, that there were significant differences in MSPAP school 
performance in 1997 and in MSPAP gains from 1993-97. In addition, the percent of students receiving 
free or reduced lunch (i.e., proxy for socio-economic status) was negatively related to MSPAP 
performance in 1997, but it was not significantiy related to MSPAP performance gains from 1993-97. 
This indicates that although schools with high levels of percent free or reduced lunch performed poorer 
on MSPAP in 1997 than schools with low levels of percent free or reduced lunch, the schools have 
similar MSPAP mathematics performance gains regardless of the percent of students receiving free or 
reduced lunch. Lastly, higher levels of teacher-reported “MSPAP Influence on Instruction” were 
associated with higher levels of rate of change in MSPAP mathematics performance over time. Thus, the 
schools for which teachers reported that MSPAP had a greater influence on their instruction had greater 
MSPAP mathematics performance gains. It is important to note that the school sample size for this 
analysis was relatively small (n=86) and therefore, the results should be interpreted cautiously. Growth 
model analyses using larger data sets may provide more stable results than the analysis in this present 
study. 
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II. Objective 



A number of states are implementing statewide assessment programs that depend heavily on 
performance-based assessments. These assessments are considered critical tools in the educational 
reform movement (Linn, 1993) and are being used for high-stakes purposes such as holding schools 
accountable to state standards. A prevailing assumption underlying performance-based assessments is 
that they serve as motivators in improving student achievement and learning, and that they encourage 
instructional strategies and techniques that foster reasoning, problem solving, and communication 
(Frederiksen & Collins, 1989; National Council on Education Standards and Testing, 1992). 

Given these high expectations for performance-based assessments, the consequences of the uses and 
interpretations of the assessments need to be addressed, including both negative and positive 
consequences, intended and plausible unintended consequences (Messick, 1989, 1992; Cronbach, 1988; 
Koretz, Barron, Mitchell, & Stecher, 1996; Linn, Baker, & Dunbar, 1991). As stated by Linn (1994), “If 
the argument that validation should include an evaluation of the consequences of the uses and 
interpretations of assessment results is accepted, then it is not sufficient to provide evidence that the 
assessments are measuring the intended constructs. Evidence is also needed that the uses and 
interpretations are contributing to enhanced student achievement and at the same time, not producing 
unintended negative outcomes (p. 8).” Further, the consideration of potentially negative effects through 
the eyes of multiple stakepersons may help ensure a more comprehensive evaluation of the consequences 
(Cronbach, 1989). 

The purpose of the study is to examine the impacts of the Maryland State Performance Assessment 
Program (MSPAP) and the Maryland Learning Outcomes (MLO’s) on school curriculum, classroom 
instruction and assessment practices, student learning, professional development, and school decision- 
making policies. MSPAP is a performance assessment program for grades 3, 5, and 8 designed to 
measure school performance and provide information for school accountability and improvement so as to 
ensure quality education (Maryland State Board of Education, 1995). MSPAP was implemented in the 
earlier 1990’s to assess student achievement and school performance with respect to the Maryland 
Learning Outcomes. MSPAP requires students to develop written responses to interdisciplinary tasks 
that require the application of skills and knowledge to real life problems. It is intended to promote 
performance-based instruction and classroom assessments. 

This study examines the impact of MSPAP for the 1996-97 instructional year for the mathematics 
and language arts content areas in elementary and middle schools in Maryland. The schools were 
classified according to the percentage of students who were receiving a free or reduced lunch, which 
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served as a proxy for socioeconomic level, and according to the school’s performance gains on MSPAP . 
The differences in the nature and die extent of die consequences of the assessment program for students 
in grade levels being tested, “on-grades” (3, 5, and 8), versus students in grade levels not being tested, 
“off-grades” (2, 4, and 7), were examined. If the intent of the assessment program is to improve student 
laming for all students regardless if they are being “tested”, then it is necessary to examine the 
' consequences for all students. 

This report includes die results for the principal, teacher, and student mathematics questionnaires as 
well as the results from the analysis of the mathematics classroom materials that were provided by 
teachers. The last section of the report discusses the relationship among MSPAP school performance 
gains and several variables such as teacher reported impact of MSPAP on instruction, teacher reported 
professional development support, and % free or reduced lunch. 
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III. Data Source 



School Sample 

A stratified random sampling procedure was used to select the schools, with die strata being defined 
by three levels of each of die following: (a) percent free or reduced lunch according to the 1994-95 
classification and (b) MSPAP performance gains (MSDE’s 1993-95 change index). Schools were 
classified into one of the nine ceils based on their rankings in the distributions for these two variables. 
Eight elementary schools from each of the nine cells were sampled and four middle schools from each of 
the nine cells were sampled. A total of 72 elementary and 36 middle schools were selected to participate 
in the study with alternate schools identified as potential replacements for schools who chose not to 
participate. A larger number of elementary schools were selected because, compared to the middle 
schools, they have fewer teachers pier grade. 

The final sample consisted of 59 elementary and 31 middle schools, with a total of 90 schools. Thus, 
the school participation rate was 82% for elementary schools and 86 % for middle schools. There was an 
approximately equal number of schools within each of the nine classification cells i. Of the 59 
elementary schools, 42 were from the initial 72 that were sampled, and of the 31 middle schools, 22 were 
from the initial 36 that were sampled. The remaining schools were from die list of alternate schools for 
each cell. This represents schools from 19 systems/counties in Maryland. It should be noted that 
because schools were unable to be contacted until January 1997 regarding their participiation in the study, 
the sample size for the 1 996-97 instructional year was reduced. 

Instruments 

To triangulate on the consequences of MSPAP, multiple measures were used. The data sources used 
for this study were questionnaires and samples of classroom instruction and assessment mator inlg 
Questionnaires were developed for principals, teachers and students. The questionnaire for principals 
was die same for both elementary and middle school principals. Separate language arts and mathematics 
questionnaires were developed for 2 nd , 3«i, 4 *, 5 th 7 th and gth grade teachers in those respective 
areas. The teacher questionnaires did not vary substantially across on- and off- grades (i.e., tested and 

1 The correlations between change indexes from consecutive years are close to zero; therefore, the cells in which the schools 
were classified changed when they were reclassified using the change index based on the 1995-% year. Schools were 
reclassified once the 1995-96 data were available because that was the year in which the data were collected for this study. It 
should be noted, however, that once the schools were reclassified there was still approximately an equal number of schools in 
each cell. 
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not tested grades, respectively). Language arts and mathematics questionnaires were developed for 
students in 5*h, 7^*, and 8^ grades. The questionnaires for the 4*h and 7^ grade (i.e., off-grade) 
students contained a MSPAP public release task so that die students could examine the task prior to 
responding to questions pertaining to MSPAP-like tasks. 

The questio nnair es consisted of both likert and constructed response items. Some of the likert items 
were in the form of questions, and others were statements. In general, a four-point scale was used for the 
likert items. To triang ulate on the consequential evidence, students, teachers, and principals responded 
to similar questions for areas in which it was deemed appropriate. The areas on the teacher questionnaire 
, included the following: familiarity with MSPAP, support for MSPAP, beliefs about MSPAP, overall 

impact of MSPAP, the nature of instruction and classroom assessments, MSPAP’ s impact on instruction 
and classroom assessments, die nature of professional development activities, and MSPAP’s impact on 
professional development activities. The principal and student questionnaires included items for areas 

I 

' that were deemed appropriate. Some of the ideas for questions pertaining to the support for MSPAP and 

the beliefs about MSPAP were based on a previous study examining the consequential evidence of state 
assessments (Koretz, Mitchell, Baron, & Keith, 1996). The instruments were piloted in the spring of 
1996 in schools in Maryland and were reviewed by Maryland mathematics teachers. 

Data collection forms were developed for a subset of the teachers in both die off-grades (2 Q d, 4th, 

7th) and on-grades (3^, 5th, 8th) who provided classroom materials. Teachers were asked to provide 10 
j instruction tasks and 10 assessment tasks that were representative of their classroom materials across the 

school year. They were also asked to provide an example scoring scheme, and an example test 
> preparation activity. The data collection forms asked teachers to indicated the nature of the students’ 

ability levels for the mathematics class that the materials were used in. (e.g., heterogeneous ability 
! group, homogeneous ability group, exclusively special education). The forms also asked teachers to 

indicate the nature of the mathematics taught in the class (e.g., general math, pre-algebra, algebra). 

t 

Data Collection 

Teachers and principals were asked to complete their respective questionnaires during February 
1997. Students were administered the student questionnaire within the two weeks following the 
| adminis tration of MSPAP, that is, in either the 3 r ^ or 4^* week of May 1997. 

Teachers were asked to send in approximately 5 mathematics instruction activities, 5 mathematics 
assessment activities, and 1 sample of a scoring scheme used from September to December 1996. 
Similarly, in the spring they were asked to send in another set of 5 instruction activities, 5 assessment 
activities, and 1 sample of a scoring scheme used from January to June 1997. In addition, they were also 
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asked to send a sample of a MSPAP test preparation activity used prior to the administration of MSPAP. 
If the teacher taught more than one mathematics class, they were requested to obtain these materials from 
a typical class that they taught 



Questionnaire and Classroom Materials Return Rate 



Principal and Teacher Questionnaire . A total of 86 of the 90 principals completed the principal 
questionnaire, resulting in a total response rate of 96%. A total of 1020 2 n< L 3 r< L 4^, 5^, 7^, and 8^ 
grade teachers out of 1 1 87 completed the teacher questionnaires, with a total response rate of 86%. Table 
m.l indicates the return rates for principals, math teachers, and language arts teachers in elementary and 
middle schools. 



Table HI. 1 Principal and Teacher Questionnaire Return Rate 





Completed Questionnaire 


Return 




Actual 


Proposed 


Rate 


Principals 








Total 


86 


90 


96% 


Elementary 


58 


59 


98% 


Middle 


28 


31 


90% 


Mathematics Teachers 








Total 


515 


594 


87% 


Elementary 


353 


405 


87% 


Middle 


162 


189 


86% 


Language Arts 
Teachers 








Total 


505 


593 


85% 


Elementary 


321 


372 


86% 


Middle 


184 


221 


83% 



Table III .2 provides the number of mathematics teachers and language arts teachers in each grade 
level who completed the questionnaires. 
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Table m.2 Number of Teachers at Each Grade Level who Completed the Questionnaire 





Grade 




2 nd 


3Hl 


4 th 


5™ 


7 tn 


8 th 


Mathematics Teachers 


79 


98 


77 


99 


62 


100 


Language Arts Teachers 


69 


90 


70 


92 


73 


111 



Student Onestionnaire . Each of the 4 th, 5 th, 7 th, and 8 th grade teachers participating in the study 
were asked to adminis ter the student questionnaire to one of their classes. Of the 637 classes that were 
identified for the admini stration of the language arts and mathematics student questionnaires, 443 classes 
completed and returned the student questionnaires, resulting in a return rate of 70%. 

Overall, 115 of the 163 elementary classes (4th and 5 th grades) that were identified for the 
administration of the mathematics student questionnaires actually administered the questionnaires, 
resulting in a return rate of 71%. In the middle school classes ( 7 th and 8 th grades), 95 of the 148 
identified classes administered the mathematics questionnaires (64%). For the language arts 
questionnaires, 115 of the 156 identified elementary classes administered the questionnaires (74%), and 
118 of the 170 identified middle school classes administered the questionnaires (69%). Table m.3 
indicates the number of students and classes in 4 th, 5 th, 7 th, and 8 th grades who completed the 
mathematics questionnaires and die language arts questionnaires. It should be noted that each of the 
questionnaires were divided into 3 forms and a student only received one form. The forms were 
randomly distributed within each of the participating classrooms. This sampling design was used to 
reduce the amount of time taken away from instruction 



Table III .3 Student Questionnaire Return Rate 





# of students 


# of classes 


# of students 
perform 


Math Students 








4 th 


1076 


48 


359 


5th 


1442 


67 


481 


7® 


845 


37 


282 


8 ® 


1207 


58 


402 


Language Arts Students 








4 tn 


1197 


54 


399 


5th 


1291 


61 


430 


7 th 


1201 


53 


400 


pi 


1358 


65 


453 
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Classroom Activities . A subset of schools were asked to participate in the collection of the 
classroom activities. Overall, 51 schools with a total of 704 teachers were asked to participate in this 
aspect of the study. Some or all of the teachers from 44 of die schools participated, resulting in a school 
participation rate for classroom activities of 86%. This represents schools from 15 different 
system/counties in Maryland. In these 44 schools, 529 of the teachers sent in all or a subset of the 
requested materials (75%). Of the 332 mathematics teachers that were asked to participate, 250 provided 
us with the materials (75%), and of the 372 language arts teachers that were asked to participate, 279 
provided us with all or a subset of the materials (75%). More information is provided in the section 
“Description of Classes and Teachers Who Collected Classroom Activities”. 



IV. Description of Principals, Teachers, and Students 



Principals and teachers responded to questions on the questionnaire regarding their experience, 
education, and gender. Students responded to questions regarding their gender and ethnicity. 

Principals 

Principals were asked to indicate the number of years they had served as a principal in their current 
school. As indicated in Table IV. 1, a smaller percentage of middle school principals were new to die 
school (1 year or less) compared to elementary principals, 15% and 26% respectively. Also, 37% of die 
middle school principals were in the school more than 6 years, while only 26% of the elementary 
principals were in the school this long. When asked if they previously held another administrative 
position in the school, more middle school principals (41%) than elementary school principals (31%) 
indicated that they did. 



Table IV. 1 Number of Years as a Principal in the Current School 





Elementary 

(n=58) 


Middle 

(n=28) 


1 year or less 


26% 


15% 


2-3 years 


21% 


22% 


4-5 years 


28% 


26% 


6-15 years 


24% 


33% 


16-25 years 


2% 


4% 


26 years or more 


0% 


0% 



Principals also provided information regarding the total number of years they had served as an 
administrator in a Maryland elementary or middle school. As indicated in Table IV .2 below, a larger 
percentage of middle school principals have had more years of experience in Maryland schools than 
elementary principals. About 30% of middle school principals compared to 14% of elementary 
principals indicated more than 1 6 years as an administrator in Maryland. 
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Table IV 2 Number of Years as an Administrator in a 
Maryland Elementary or Middle School . 





Elementary 

(n=58) 


Middle 

(n=28) 


1 year or less 


7% 


7% 


2-3 years 


5% 


4% 


4-5 years 


16% 


19% 


6-15 years 


59% 


41% 


16-25 years 


14% 


19% 


26 years or more 


0 % 


11 % 



Teachers 

The majority of the teachers responding to the questionnaire were female. Across grades, 87% of the 
math teachers were female and 90% of die language arts teachers were female. In general, as the grade 
level increased, the percentage of male teachers increased slightly. For instance, about 21% of the 
middle school teachers responding to the mathematics questionnaire were male. 

The percentage of mathematics and language arts teachers who were new to their current school (1 
year or less) was 16% for all grades as indicated in Table IV .3. About 37% were in the school for 2 to 5 
years, 29% were in the school for 6 to IS years, and 19% were in die school more than 16 years or more. 
There were slightly more mathematics teachers who were new to the school (1 year or less) than 
language arts teachers. The percentages across grade levels were somewhat similar. 



Table IV.3 Number of Years as a Teacher in the Current School 





All 

grades 


2 nd 


3 rd 


1 ffi” 


im- 


7 th 


8 th" 


1 year or less 


16% 


12 % 


13% 


13% 


17% 


21 % 


16% 


2-3 years 


23% 


17% 


21 % 


26% 


28% 


24% 


22 % 


4-5 years 


14% 


12 % 


16% 


11 % 


14% 


15% 


15% 


6-15 years 


29% 


36% 


33% 


28% 


25% 


22 % 


29% 


16-25 years 


15% 


16% 


13% 


15% 


13% 


17% 


17% 


26 years or more 


4% 


7% 


3% 


7% 


3% 


1 % 


2 % 



Teachers also provided information regarding the total number of years they had taught in a 
Maryland school. The results for mathematics and language arts teachers were similar and are 
summarized in Table IV .4. An examination of the results at each grade level indicate slightly larger 
percentages of new teachers in the middle school grades. 





Table IV.4 Number of Years as a Teacher in a 
Maryland Elementary or Middle School 





All 

grades 


2 nd 


3 rd 


4 th 


5 th 


7 th 


8 th 


1 year or less 


8 % 


6 % 


6 % 


6 % 


8 % 


11 % 


10 % 


2-3 years 


15% 


10 % 


18% 


15% 


18% 


18% 


12 % 


4-5 years 


11 % 


9% 


18% 


10 % 


16% 


8 % 


10 % 


6-15 years 


29% 


35% 


10 % 


25% 


23% 


27% 


31% 


16-25 years 


25% 


24% 


32% 


27% 


23% 


24% 


25% 


26 years or more 


12 % 


15% 


24% 


17% 


11 % 


11 % 


11 % 



The vast majority of the mathematics and language arts teachers (98%) indicated that they were full- 
time teachers. Mathematics teachers were then asked to indicate whether they taught only math classes. 
Similarly, language arts teachers were asked to indicate whether they taught only language arts. As 
expected, there was a large difference among the grades as shown in Table IV.5. The majority of the 
middle school teachers taught only one subject, while the majority of the elementary school teachers 
taught more than one subject Language arts teachers were also asked to indicate whether they taught 
reading and writing, reading only, or writing only. Nearly all the teachers (90%) indicated that they 
taught both reading and writing. The results were similar across all grade levels. 



Table IV.5 Percentages of Teachers Indicating That They Taught 
Only Mathematics or Language Arts Classes 



Mathematics Teachers 


All 

grades 


2 nd 


3 rd 


4 th 


5th 


7 th 


8 th 


Teach only math 


29% 


3% 


8 % 


7% 


10 % 


77% 


70% 


Teach math and other 
subjects 


72% 


97% 


92% 


93% 


90% 


23% 


30% 


Language Arts Teachers 
















Teach only language arts 


36% 


12 % 


13% 


10 % 


20 % 


73% 


76% 


Teach language arts and 
other subjects 


64% 


88 % 


87% 


90% 


81% 


28% 


24% 



Each teacher responded to one questionnaire that was specific to a subject area (mathematics or 
language arts) and a grade level (2, 3, 4, 5, 7, or 8 ). They were asked to indicate the number of 
“targeted” classes they taught. “Targeted” refers to classes in the subject area and grade level for which 
they received a questionnaire. Table IV .6 shows that the majority of die elementary teachers taught only 



one “targeted” class, while a large percentage of middle school teachers, especially 7 ^ grade teachers, 
taught 4 or more “targeted” classes (62% for grade mathematics teachers and 58% of 7^* grade 
language arts teachers). 



Table IV .6 Percentages of Mathematics and I -antniage Arts Teachers 
Indicating the Number of ‘Targeted” Classes Taught 



Mathematics Teachers 


All 

grades 


2 nd 


3 rd 


4 th 


5 th 


7 th 


"jar 


1 class 


67% 


93% 


90% 


76% 


68% 


18% 


26% 


2-3 classes 


19% 


4% 


7% 


19% 


26% 


20 % 


29% 


4 or more classes 


21 % 


3% 


2 % 


1 % 


6% 


62% 


45% 


Language Arts Teachers 
















1 class 


51%. 


79% 


78% 


74% 


58% 


15% 


17% 


2-3 classes 


23% 


16% 


13% 


13% 


32% 


27% 


32% 


4 or more classes 


23% 


2 % 


5% 


8% 


10 % 


58% 


49% 



Mathematics and language arts teachers were asked to indicate the type of teaching certificate they 
hold. The majority of elementary teachers in both subject areas have an elementary education certificate, 
however the percentage is much smaller for middle school teachers as indicated in Table IV.7. About 
44% of 7th grade mathematics teachers and 55% of 8 ^ grade mathematics teachers have some type of 
secondary education certificate in mathematics. Likewise, about 56% of 7 *h grade language arts teachers 
and 58% of 8 ^ grade language arts teachers have some type of secondary education certificate in 
language arts or English. The types of certificates included in the “other” category were special 
education, administration/supervision, and non-education content areas such as mathematics, science, 
etc. 
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Table IV.7 Type of Teaching Certificate Held bv Mathematics and Language Arts Teachers 



Mathematics Teachers 


All 

grades 


2nd 


31X1 


4ttl 


5*5“ 


“ 7tE~ 


gffi" 


Early Childhood 


12% 


42% 


11% 


10% 


4% 


2% 


3% 


Elementary Education 


70% 


68% 


85% 


92% 


90% 


48% 


33% 


Secondary Education in 
Mathematics, 5-12 


10% 


0% 


1% 


0% 


2% 


29% 


30% 


Secondary Education in 
Mathematics, 7-12 


7% 


0% 


1% 


0% 


1% 


15% 


25% 


Secondary Education in 
another subject 


5% 


0% 


3% 


3% 


2% 


11% 


12% 


Other 


18% 


10% 


12% 


16% 


18% 


23% 


27% 


Language Arts Teachers 
















Early Childhood 


12% 


35% 


23% 


9% 


9% 


0% 


1% 


Elementary Education 


63% 


74% 


74% 


90% 


89% 


29% 


32% 


Secondary Education in 
Language Arts/English, 
5-12 


15% 


0% 


2% 


6% 


4% 


30% 


37% 


Secondary Education in 
Language Arts/English, 
7-12 


9% 


0% 


1% 


3% 


1% 


26% 


21% 


Secondary Education in 
another subject 


5% 


0% 


2% 


1% 


4% 


11%. 


10% 


Other 


19% 


13% 


18% 


19% 


23% 


16% 


22% 



Finally, mathematics and language aits teachers were asked to indicate whether they had ever been a 
Reader for MSPAP and if they had ever been on their School Improvement Team. Approximately 25% 
of the teachers indicated that they were served as a Reader for MSPAP on one or more occasions since 
1992. Slightly more than 50% of die teachers indicated that they were on their School Improvement 
Team for at least one school year since 1992-93. 



Students 

Approximately 50% of die students responding to the mathematics and language arts questionnaires 
were female and 50% were male. This was similar across all grade levels. Students were also asked to 
indicate their ethnicity. As Table IV.8 shows, die majority of students (about 70%) indicated Caucasian, 
approximately 20% indicated African-American, and a very small percentage indicated Hispanic, Asian 
American, or other. 
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Table IV .8 Ethnicity of Students Responding to the Questionnaire 



Mathematics Students 


All 

grades 




5 th 


7 th 


”8*5“ 


White 


70% 


67% 


65% 


73% 


75% 


African-American 


17% 


16% 


19% 


17% 


15% 


Hispanic 


4% 


3% 


4% 


3% 


4% l 


Asian-American 


5% 


7% 


7% 


3% 


2 % 


Other 


5% 


7% 


5% 


4% 


4% 1 


Language Arts Students 












White 


67% 


69% 


58% 


72% 


70% 


African-American 


21 % 


19% 


27% 


19% 


19% 


Hispanic 


3% 


2 % 


3% 


2 % 


5% 


Asian-American 


3% 


5% 


5% 


2 % 


2 % 


Other 


5% 


6 % 


7% 


4% 


4% 
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V. Descriptive Results from Principal, Teacher, and Student 

I 

Mathematics Questionnaires 



The results are provided in several sections: Familiarity with MSPAP, Support for MSPAP, Beliefs 
about MSPAP, and the Impact of MSPAP. For the section on die Impact of MSPAP, the results are 
further or ganize d into the following sections: Overall Impact of MSPAP, Impact of MSPAP on 

Instruction and Classroom Assessment, and the Impact of MSPAP on Professional Development 
Activities. The results from the principal questionnaires are from all participating principals. However, 
the results from the teacher questionnaires are from those teachers who received a mathematics teacher 
questionnaire and the results from die student questionnaires are from those students who received a 
mat h emat ics student questionnaire. The results from the language arts questionnaires are forthcoming. 



Familiarity with MSPAP and Related Materials 

Principals and teachers were asked to indicate the extent to which they were familiar with the 
Maryland Learning Outcomes (MLO’s), Maryland Curriculum Framework, and various aspects of 
MSPAP, including the purpose of MSPAP, format of MSPAP tasks, content and skills assessed by 
MSPAP, how to prepare students for MSPAP, how to interpret and use MSPAP results to improve 
classroom instruction and assessment, and how to explain MSPAP results to students and/or parents. 
They were also asked to indicate their familiarity with several MSPAP-related materials and documents, 
including MSPAP public release tasks, Performance-Based School Improvement Exemplars. Teacher-to- 
Teacher Talk. Princinal-to-Principal Talk. Scoring MSPAP: A Teacher’s Guide. MSPAP Parent 

Handbook and MSPAP: A Guide for Parents . Lastly, students were asked if they knew the purpose of 
MSPAP and if they knew what types of answers to math tasks on MSPAP would receive high scores. 



Familiarity with the MLO’s and Maryland Curriculum Framework 

Most of the principals indicated that they had either a moderate amount or a great deal of familiarity 
with tiie Maryland Learning Outcomes (93%) and the Maryland Curriculum Framework (81%). 
Teachers indicated less familiarity than principals with the MLO’s and the Maryland Curriculum 
Framework. The percentage of teachers that indicated they had either a moderate or a great extent of 
familiarity with the MLO’s was 66% and with the Maryland Curriculum Framework it was 54%. 
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Middle school and elementary school principals reported similar levels of familiar ity with the 
MLO’s and Maryland Curriculum Framework. Teachers across grades also reported similar levels of 
familiarity with the Maryland Curriculum Framework. With respect to die MLO’s, however, 74% of the 
3 r ^ and 5*h grade teachers reported being familiar with the MLO’s to a moderate or great extent, whereas 
teachers in the other grade-levels (2®d, 4^, 7^, and 8^) reported less familiarity. The 7^* grade teachers 
reported the least amount of familiarity, with 45% indicating a moderate or great extent of familiarity. 

Familiarity with MSPAP 

Purpose, Format, and Content of MSPAP . With respect to the familiarity with MSPAP, nearly all 
the principals indicated that they had a moderate amount or a great deal of familiarity with the purpose of 
MSPAP (99%), the format of MSPAP tasks (99%), and the content and skills assessed by MSPAP 
(96%). In addition, most of the teachers indicated that they had a moderate or great extent of familiarity 
with the purpose of MSPAP (93%), the format of MSPAP tasks (90%), and the content and skills 
assessed by MSPAP (86%). 

Preparing Students for MSPAP . Most of the principals indicated that they know how to prepare 
students for MSPAP to a moderate or great extent (98%). Similarly, 85% of the teachers indicated that 
they know how to prepare students for MSPAP to a moderate or a great extent. 

MSPAP M aterials and Publications . The majority of principals indicated that they had either a 
moderate or great extent of familiarity with MSPAP materials and publications. Depending upon the 
particular publication or set of materials, between 85% and 98% of the principals indicated a moderate or 
great deal of familiarity. Overall, a smaller percentage of teachers were familiar with the various 
MSPAP publications and related materials. Over half of the teachers were familiar with the MSPAP 
public release tasks (60%), MSPAP Performance-Based School Improvement Exemplars (56%), and 
MSPAP T eacher-to-T eacher Talk (62%). Whereas, less than half of the teachers were familiar with 
Scoring MSPAP: A Teacher’s Guide (43%) , MSPAP Parent Handbook (33%), and MSPAP: A Guide 
for Parents (33%). 

Results of MSPAP . With respect to the use of the MSPAP results, the majority of the principals 
indicated that, to a moderate or to a great extent, they know how to interpret and use MSPAP results to 
improve classroom instruction and assessment (92%), and how to explain MSPAP results to students 
and/or parents (91%). Teachers, however, reported less familiarity than principals with knowing how to 
interpret and use MSPAP results: 68% indicated that they had a moderate or a great extent of familiarity 
with interpreting and using MSPAP results to improve classroom instruction, and 53% indicated that 
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they had a moderate or a great extent of familiarity with explaining MSPAP results to students and/or 
parents. 

Principals and teachers were also asked to indicate the format in which MSPAP results are shared 
with teachers in their school, whether it be results from the Maryland School Performance Reports. 
Proficiency Level and Participation Reports. Outcome Score and Scale Score Reports, locally developed 
MSPAP reports, or other types of reports. Table V.l shows the percentages of principals and teachers 
who indicated the various formats used for informing teachers of MSPAP results. 
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Table V. 1 Percentage of Principals and Teachers Indicating die Formats for 
Informing Teachers of MSPAP Results 



Formats for Sharing MSPAP Results 


Principals 


Teachers 


Teachers have die opportunity to see actual written reports 


90% 


52% 


Principals or others provide written summaries of reports 


85% 


80% 


Principals or others provide verbal summaries of reports 


94% 


78% 


MSPAP results are not shared with teachers 


1% 


2% 



Overall, principals and teachers indicated that teachers are aware of the results of MSPAP. Only 1% 
and 2%, respectively, reported that the MSPAP results are not shared with teachers. A large percentage 
of both principals and teachers indicated that the principal or someone else provides teachers with 
written and verbal summaries of MSPAP results. However, in terms of teachers seeing the “actual 
written reports”, principals and teachers have differing views. Most of the principals (90%) said that 
teachers have the opportunity to see actual reports, while only half ( 52 %) of the teachers reported being 
aware of the opportunity to see actual reports. 

Differences Between Elementary and Middle Sc hnnl Principals. There were several differences 
between elementary and middle school principals’ reported level of familiarity with MSPAP. Overall, 
the elementary school principals reported being more familiar than the middle school principals. The 
differences were the greatest for the following aspects of MSPAP: 

1. Content and skills assessed bv MSPAP : 

67% of elementary versus 37% of middle school principals were familiar to a “great extent”. 

2. MSPAP Public release tasks : 

All elementary versus 81% of middle school principals were familiar to a “moderate” or “great 
extent”. 

3. MSPAP Performance Based School Improvement Exemplars : 

95% of elementary versus 74% of middle school principals were familiar to a “moderate” or 
“great extent” ’. 
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4. Scoring MSPAP: A Teacher’s Guide : 

53% of elementary versus 22% of middle school principals were familiar to a “great extent”. 

Whereas, 10% of elementary and 26% of middle school principals were “not at all” familiar . 

Differences Across Grade-Level Teachers . Teachers across grades tended to report similar levels 
of familiarity with respect to the purpose of MSPAP. However, differences by grade-level occurred for 
the other aspects of MSPAP. The 3 r< ^ and 5^ grade teachers (elementary, on-grade) tended to be the 
most familiar with the format of MSPAP, content and skills assessed by MSPAP, and preparation of 
students for MSPAP. Over 90% of the and 5^ grade teachers reported being familiar with these 
aspects to a moderate or great extent Whereas, the 7^ grade teachers tended to be the least familiar 
with these aspects of MSPAP. On average, about 70% of the 7th grade teachers reported a moderate or 
great deal of familiarity. 

Differences between these grade levels were the greatest for die use of MSPAP results and 
documents related to MSPAP. Approximately 25% to 33% of the 7th grade teachers reported that to a 
moderate or great extent they were familiar with diem, whereas 50% to 75% of the 3”* and 5th grade 
teachers reported this level of familiarity. 

Student Familiarity with MSPAP 

Students were asked if they knew why MSPAP was given to students. Overall, 65% of die students 
said “yes” and 35% said “no”. However, there were differences in the responses between the off-grade 
students (grades 4 and 7) and the on-grade students (grades 5 and 8). A larger percentage of 5*h and 8*h 
grade students said they knew why MSPAP was given to students (appro ximate ly 72%) compared to 
about 54% of die 4^h and 7*h grade students. This question also asked students to explain why they think 
MSPAP is given to students if they answered “yes”. 

The 5th and 8^ grade students were also asked if they knew what types of answers to math tasks on 
MSPAP would receive good grades. Overall, 57% said “yes” and 43% said “no”. The percentage of 5^ 
grade students who indicated “yes” was slighdy larger than 8^ grade (61% and 51%, respectively). 
Students were also asked to describe what types of responses would get good grades. 

The 4*h and 7*h grade students were asked if they knew what types of math tasks were on MSPAP 
before they saw the MSPAP example task in their questionnaire. Overall, 40% of the students said “yes” 
and 60% said “no”. 
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Support for MSPAP 



Principals and teachers were asked to indicate the extent to which they support MSPAP overall, the 
extent to which they support using MSPAP results for accountability purposes, and die extent to which 
they support the use of MSPAP as a tool for instructional change. 

Overall Support for MSPAP 

Most of the principals indicated that they either somewhat or strongly support MSPAP (87%) and 
that they have become more supportive over the years (86%). Although the teachers* percentages are 
slighdy smaller than the principals’ percentages, the majority of elementary and middle school teachers 
indicated that they either somewhat or strongly support MSPAP (71%) and that they have become more 
supportive of MSPAP over the years (65%). 

Principals and teachers were also asked whether they agreed with the statement that MSPAP results 
provide useful information for making inferences about school improvement. A larger percentage of 
principals, compared to teachers, reported that they somewhat or strongly agreed with this statement, 
82% and 56% respectively. There were no differences in the percentages across grade levels nor 
between type of school. 

Principals were also asked whether the School Improvement Team (SIT) uses information provided 
by MSPAP when setting forth goals for their schools and for helping improve the quality of instruction 
in heir schools. Most of the elementary and middle school principals somewhat or strongly agreed (95% 
for both). However, a larger percentage of elementary principals, as compared to middle school 
principals, “strongly” agreed with the statement regarding the SIT using MSPAP information to help 
improve the quality of instruction in their schools (64% and 36%, respectively). 

Use of MSPAP Results for Accountability Purposes 

Table V.2 indicates the level of principal and teacher support for using MSPAP results for 
accountability purposes. Only the overall levels of support are included since the results were similar for 
elementary and middle school principals and also across grade-levels for teachers. As indicated in the 
table, principals, as compared to teachers, reported being more supportive of using MSPAP for 
accountability purposes. Although over half of the teachers somewhat or strongly support tire reporting 
of MSPAP results, the majority of the teachers somewhat or strongly oppose holding schools accountable 
for meeting the performance standards on MSPAP, using MSPAP and other report card results for 
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identifying schools as eligible for reconstitution, and using MSPAP and other report card results for 
identifying schools as eligible for recognition or monetary rewards. 



Table V.2 Percentage of Principals and Teachers who Support 
MSPAP for Accountability Purposes 





Strongly or 
Somewhat Support 


Strongly or 
Somewhat Oppose 


Principals 


Teachers 


Principals 


Teachers 


Reporting of MSPAP results 


71% 


53% 


29% 


47% 


Holding schools accountable for meeting the 
performance standards on MSPAP 


79% 


37% 


21% 


67% 


Using MSPAP and other report card results for 
identifying schools as eligible for reconstitution 


54% 


26% 


46% 


73% 


Using MSPAP and other report card results for 
identifying schools as eligible for recognition or 
monetary rewards 


68% 


25% 


33% 


75% 



MSPAP as a Tool for Instructional Change 

As indicated in Table V.3, nearly all of the principals (94%) somewhat or strongly agreed with the 
statement that MSPAP is a useful tool for making positive changes in instruction, and 78% of the 
principals somewhat or strongly agreed with the statement that MSPAP is a useful tool for making 
positive changes in instruction for those teachers who are resistant to change. Elementary and middle 
school principals responded similarly to these questions. 



Table V.3 Percentage of Principals and Teachers who Support 
MSPAP as a Tool for Instructional Change 





Strongly or 
Somewhat Support 


Strongly or 
Somewhat Oppose 


Principals 


Teachers 


Principals 


Teachers 


Useful tool for making positive changes in 
instruction 


94% 


77% 


6% 


23% 


Useful tool for making positive changes in 
instruction for those teachers who are 
resistant to change 


78% 


57% 


12% 


43% 



Teachers indicated that they were much more supportive of MSPAP for instructional purposes than 
for accountability purposes. The majority of the teachers (77%) somewhat or strongly agreed with the 
statement that MSPAP was a useful tool for makin g positive changes in their own instruction and 57% of 
the teachers somewhat or strongly agreed with die statement that MSPAP was a useful tool for making 
positive changes in instruction for those teachers who are resistant to change. Teachers across grades 
tended to respond similarly to questions regarding their support for MSPAP as an instructional tool. 



Beliefs About MSPAP 



Principals and teachers were asked questions regarding their beliefs about the thinking skills assessed 
by MSPAP, the developmental appropriateness of MSPAP, die feasibility of meeting MSPAP standards, 
and the factors that affect MSPAP results. Teachers and students were asked questions regarding their 
beliefs about MSPAP tasks, multiple-choice items, and the expected level of student performance on 
MSPAP. 

Beliefs About the Skills Assessed by MSPAP 

Principals and teachers were asked to indicate the extent to which they think MSPAP assesses 
mathematical procedures/factual knowledge and mathematical problem solving/reasoning. In addition, 
they were asked to indicate whether they think MSPAP should place more or less emphasis on these 
types of thinking skills. Overall results for principals and teachers are in Table V.4. 



Table V.4 Percentages of Principals and Teachers Indicating Their Beliefs About the 

Thinking Skills MSPAP Assesses 



Emphasis on Thinking Skills 


Moderate 

Extent 


Great 

Extent 


Principals 


Teachers 


Principals 


Teachers 


Math procedures & factual knowledge 


54% 


76% 


26% 


14% 


Math problem solving & reasoning 


28% 


28% 


68% 


51% 



Appropriateness of MSPAP’s Emphasis 


Emphasis Is 
OK 


Needs 

Emp 


More 

lasis 


Principals 


Teachers 


Principals 


Teachers 


Math procedures & factual knowledge 


70% 


42% 


21% 


33% 


Math problem solving & reasoning 


73% 


62% 


12% 


12% 



The majority of principals and teachers indicated that MSPAP assesses both of these types of skills 
to either a moderate or great extent. However, as shown in the table, a relatively large percentage of 
principals and teachers (68% and 51%, respectively) think that MSPAP assesses problem solving and 
reasoning to a great extent, whereas only 26% and 14% of principals and teachers think that MSPAP 
assesses procedures and factual knowledge to a great extent 

Approximately 70% of the principals indicated that the emphasis on die two types of thinking skills 
was appropriate. A smaller percentage of teachers felt this way, especially for die emphasis on 
procedures and factual knowledge. Approximately 42% of the teachers said the emphasis was 
appropriate, whereas 33% indicated that more emphasis was needed. 

Differences Across Grade Levels . Elementary and middle school principals reported similar beliefs 
about the extent to which MSPAP assesses procedures and factual knowledge, but they tended to differ 
in terms of MSPAP’ s assessment of problem solving and reasoning. About 78% of elementary 
principals reported that MSPAP assesses problem solving and reasoning to a moderate or great extent, 
compared to only 54% of the middle school principals. In addition, fewer elementary (7%) than middle 
school principals (23%) reported that MSPAP should place more emphasis on problem solving and 
reasoning. 

There were no differences across grades for teachers with the exception of the percentage of teachers 
who selected the “don’t know” option for these questions. A higher percentage of off-grade teachers (2, 
4, 7) than on-grade teachers (3, 5, 8) reported that they did not know the extent to which either of these 
types of skills were assessed. For instance, almost 30% of the 2°d and 4th grade teachers said “don’t 
know” compared to about 5% of the on-grade teachers. Thus, as might be expected, off-grade teachers 
have less familiarity with the types of skills assessed by MSPAP. 

Beliefs About the Developmental Appropriateness of MSPAP 

Principals and teachers were asked to indicate the extent to which they agree or disagree with the 
statement that the mathematics tasks on MSPAP are appropriate for students’ ability levels. The 
elementary principals indicated the app ro p riateness of Grade 3 and Grade 5 MSPAP tasks and middle 
school principals indicated the appropriateness of Grade 8 MSPAP tasks. The 2 Q d and 3*^ grade 
teachers indicated the appropriateness of Grade 3 MSPAP tasks, 4th and 5th grade teachers indicated the 
appropriateness of Grade 5 MSPAP tasks, and 7th and 8th grade teachers indicated the appropriateness 
of Grade 8 MSPAP tasks. The percentages in Table V.5 represent the combination of the “somewhat” 
and “strongly” agree categories. 
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Table V.5 Percentages of Principals and Teachers Indicating the Appropriateness of 
MSPAP Math Tasks for Students’ Ability Levels 



Appropriateness of MSPAP 
Math Tasks 


Somewhat/Strongly 

Agree 


Principals 


Teachers 


Grade 3 Tasks 


22 % 


2 nd 

50% 


3 rd 

35% 


GradeS Tasks 


64% 


4 th 

39% 


5 tn 

39% 


Grade 8 Tasks 


63% 


7 tn 

52% 


pE 

37% 



As the results show, principals indicated that the 5*h grade and 8 *E grade mathematics tasks on 
MSPAP were more aligned to the students’ ability level than die 3 rd grade tasks. In fact, 36% of die 
elementary principals strongly disagreed with the statement that 3 rd grade MSPAP mathematics tasks 
were appropriate for 3 rd grade students. 

With respect to the teachers, there was little variation among on-grade teachers (3 rd , 5 th , 8 th ) 
across grades: die percentages agreeing that the ability levels of the math ta sks on MSPAP are 
appropriate are between 35% and 39%. Compared to the principals’ results, the percentages of teachers 
indicating that the 5 ^ and 8 * grade MSPAP tasks were aligned to the students’ ability level were 
smaller. Further, relatively larger percentages of 2 nd and 7 th grade teachers versus 3 rd and 8 th grade 
teachers indicated that the MSPAP mathematics tasks were appropriate for students’ ability level. These 
results may be due to die off-grade teachers not being as familiar with MSPAP- like tasks as the on-grade 
teachers. This difference between off- and on-grade teachers was noted in some of die results described 
previously. 

Beliefs about MSPAP Tasks and Multiple-Choice Items 

MSPAP Tasks . Students and teachers were asked a set of four questions related to the mathematics 
tasks on MSPAP. The off-grade students (4 th and 7 th ) examined a MSPAP public release task prior to 
answering the questions. Table V .6 shows die percentages of students and teachers across all grade levels 
who indicated either “moderately” or “very” for each of the questions. It should be noted that the 
teachers responded to the questionnaire prior to die administration of MSPAP, whereas the students 
responded after the administration of MSPAP. 



Table V.6 Percentages of Students and Teachers Indicating Various 
Beliefs About MSPAP Tasks 



Aspects of MSPAP Tasks 


Moderately/V ery 


Students 


Teachers 


How interesting are foe math tasks? 


48% 


56% 


How well do they let you (your students) show what they know? 


66% 


48% 


How important is for you (your students) to do well on MSPAP?** 


89% 


46% 


How hard did you (your students) try to solve foe math tasks?** 


93% 


61% 



**The percentages in the table for these questions reflect the responses of 5 th and 8 th grade students only. 



Similar results were found across die students and teachers in terms of how interesting they think the 
tasks are, but a larger percentage of students than teachers felt that die MSPAP tasks let diem show what 
they know in math. The greatest difference between students and teachers occurred for die questions 
related to how important it is for students to do well and how hard students tried. Most of the students 
as compared to teachers indicated that it was important for them to do well and that they tried hard. 

For each of these questions, differences occurred across grade levels. In some instances, the results 
differed for elementary versus middle grades and in other instances the differences were for off-grades 
versus on-grades. Further, between 10% and 24% of foe off-grade teachers selected “don’t know” for 
this set of questions which most likely indicates that foe off-grade teachers are less familiar with foe 
types of tasks on MSPAP and/or students’ reactions to MSPAP. When comparing all foe grades, foe on- 
grade elementary students and teachers (3*d and 5*h grade) had foe largest percentages in foe 
“moderately” and ‘Very” categories, whereas foe on-grade middle school students and teachers (8*h 
grade) had foe smallest percentages in these categories. Several differences for each question are 
summarized below. 

1 . How interesting are foe math tasks for students?: 

58% of elementary students vs. 36% of middle school students indicated “moderately” or “very”. 
14% of elementary students vs. 34% of middle school students indicated “not at all”. 

There were no differences for teachers across grades. 

2. How well do they let you (your students) show what they know?: 

50% of 8*h grade students vs.73% of 4*h, 5^, and 7*h grade students indicated “moderately” or 
‘Very”. 

33% of 8th grade teachers vs. 50% of all other teachers indicated “moderately” or “very”. 

3. How important is for you (your students) to do well on MSPAP?: 

79% of 5th grade students vs. 54% of 8*h grade students indicated “very”. 

58% of 3 r£ t and 5*h grade teachers vs. 29% of 8^ grade teachers indicated “moderately” or 
“very”. 
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4. How hard did you (your students) try to solve the math tasks?: 

76% of 5 th grade students vs. 58% of 8 th grade students indicated “very”. 

71% of 3*^ and 5^ grade teachers vs. 38% of 8^ grade teachers indicated “moderately” or 

“very”. 

The 4 th 5 5 th ; 'jtia, and 8*h grade students were also asked to compare MSP AP-like tasks to multiple 
choice items with respect to a number of features. As shown in Table V.7, the majority of students 
indi cated that multiple-choice items are more interesting and that it is easier to know what to do when 
working on multiple-choice items as indicated in the table below. They also indicated that they like 
multiple-choice items better than MSPAP-like tasks. However, the majority of students reported that 
MSPAP-like tasks make you think harder and that they are better at letting you show what you know. 
Thus, even though students may prefer multiple-choice items, perhaps because they perceive them as 
easier, students believe that MSPAP tasks assess their mathematical understanding better and that they 
require higher levels of thinking skills. 



Table V.7 Percentage of Students Indicating Their Preference of Task Type 



Features of Tasks 


MSPAP 


Multiple 

Choice 


Same 


More interesting 


19% 


60% 


21% 


Makes you think harder 


83% 1 


8% 


9% 


Easier to know what to do 


10% 


79% 


12% 


Better at letting you show what you know 


67% 


15% 


18% 


Like better 


14% 


72% 


14% 



Students across grade levels responded similarly for the questions regarding the type of task that 
makes students think harder, die type of task for which it is easier to know what to do, and the type of 
task that is the most interesting. However, the responses of elementary and middle school students 
differed in terms of the type of task they like better. Overall, a larger percentage of middle school 
students (7 th and 8*) compared to elementary students (4 th and 5 th ) indicated that they liked multiple- 
choice items better than MSPAP-like tasks, 82% vs. 65% respectively. For die question regarding the 
type of task that is better at letting students show what they know about math, a smaller percentage of 8^ 
grade students (55%) compared to 4*h, 5^, and 7^ grade students (approximately 70%) said MSPAP. 
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Comparison of MSPAP to Standardized Mqltiple-Choice Tests . Mathematics teachers and 
principals were asked to indicate the extent to which they agreed with die following two statements: 
“There are advantages of MSPAP over standardized multiple-choice tests (e.g., CTBS).”, and “There are 
advantages of standardized multiple-choice tests over MSPAP.” Results showed that 77% of the 
mathematics teachers and 94% of die principals either somewhat or strongly agreed that there were 
advantages of MSPAP over standardized multiple-choice tests. Whereas, 66% of the mathematics 
teachers and 57% of the principals indicated that there are also advantages of standardized tests over 
MSPAP. 

Teachers and principals were also asked to describe the advantages, if any, of each type of test. Of 
the 515 mathematics teachers who completed the questionnaire, 336 (or 65%) provided written 
comments for this item, and 59 out of the 86 principals (69%) provided written comments. Over half of 
the teachers and principals (55% and 53%, respectively) indicated at least one advantage of MSPAP and 
at least one advantage of standardized tests. About 25% of the teachers and 34% of die principals 
commented only on the advantages of MSPAP. About 19% of the teachers and 12% of the principals 
commented only on the advantages of standardized tests. Less than 1% of the teachers’ and principals’ 
comments were too vague to interpret It should be noted that when asked to describe the advantages of 
standardized tests, the comments from teachers and principals were often worded in terms of the 
disadvantages of MSPAP. 

On average, each teacher’s and principal’s response contained approximately 3 advantages of 
MSPAP and/or advantages of standardized tests. Table V.8 includes the most frequently cited 
advantages of MSPAP. The percentages in the table reflect the percent of total comments coded. For 
example, 17% of the stated advantages of MSPAP provided by teachers were that the test requires high- 
level thinking skills, and 16% of the stated advantages of MSPAP provided by principals were related to 
high-level thinking skills. This advantage was die one most frequently cited by both principals and 
teachers. Other frequent advantages cited by teachers were that MSPAP allows students to show what 
they know or are performance-based, assesses communication, and contains applied and realistic items 
or prepares students for the future. Examples of advantages of MSPAP cited by principals include less 
memorization of facts and procedures, contains applied and realistic items or prepares students for the 
future, and less chance of guessing correctly. As can be seen by the percentages in the table, the list of 

I 

advantages was quite varied and accounts for approximately 80% of the total comments provided by 
teachers and principals. Several other advantages were noted less frequently such as allowing students to 
demonstrate their knowledge in different ways, more than one right answer to an item, integration across 
subject areas, and engaging and interesting items. 
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Table V.8 Advantages of MSPAP over Sta ndardized Tests 



Advantages of MSPAP 


Percentage of 
Teachers’ 
Comments 


Percentage of 
Principals’ 
Comments 


Requires high-level thinking skills 


17% 


16% 


Allows students to show what they know/performance-based 


16% 


10% 


Assesses communication 


12% 


6% 


Contains applied, realistic items/preparation for future/real-life 


12% 


14% 


Less emphasis on memorization of facts and procedures 


9% 


15% 


Less chance of guessing correctly 


9% 


8% 


Includes cooperative learning 


5% 


5% 



Table V.9 provides the teachers’ and principals’ responses when asked to describe die advantages of 
standardized tests. For the most part, the responses were worded in terms of the disadvantages of 
MSPAP rather than the advantages of standardized tests. Once again, the percentages in the table reflect 
the percent of total comments coded. For example, 13% of all the disadvantages of MSPAP provided by 
teachers and 9% of all the disadvantages provided by principals were that the scoring is either too 
difficult and time-consuming or that it is too subjective. The most frequently cited disadvantaged by 
principals was that individual scores are needed for student evaluations and comparisons across years or 
across schools. Some of the other disadvantages most frequently cited by both teachers and principals 
are that MSPAP removes the emphasis on basic skills, involves too much writing and/or reading, a week 
is too long for students and teachers to take a test, and results are not timely. The list in this table 
accounts for approximately 70% of all the disadvantages. Other types of disadvantages that were noted 
less frequently are that MSPAP is more stressful for students than sta nd ardized tests, it is too difficult for 
certain groups of students, it is not standardized or nationally normed, and it is too costly. 



Table V.9 Disadvantages gfMSPAP 



Disadvantages of MSPAP 


Percentage of 
Teachers’ . 
Comments 


Percentage of 
Principals’ 
Comments 


Scoring is too difficult/subjective 


[ 13% 


9% 


Removes emphasis on basic skills 


12% 


11% 


Individual scores are needed for student 
evaluation/comparisons 


9% 


16% 


Doesn’t show what student knows 


8% 


4% 


Not a good format (multiple-choice is better) 


8% 


3% 


Involves too much writing and/or reading 


11% 


5% 


Week is too long for students and teachers for testing 


6% 


9% 


Results are not timely 


2 % 


8% 


Not age-appropriate/too difficult 


5 % 


4% 



Beliefs About the Expected Level of Student Performance on MSPAP Math e ma tics T as ks 

Students in 5 th and 8* grade were asked to indicate whether they drought drey performed at the 
excellent, satisfactory, or below satisfactory levels on the MSPAP mathematics tasks. Similarly, 
teachers were asked to estimate the percentage of their students that would perform at these three levels. 
The results were similar across grade levels for both teachers and students; therefore the results in Table 
V.10 are not broken down by grade level. The percentages in the teacher col umn represent an average 
across teachers of the percentages of students that they estimated would perform at each level. 



Table V.10 Percentages of Students* and Teachers’ Estimates for 
MSPAP Performance Levels 



MSPAP Student 
Performance 


Teachers’ 

Estimate 


Students’ 

Estimate 


Excellent 


14% 


31% 


Satisfactory 


45% 


64% 


Below Satisfactory 


45% 


5% 



On average, the majority of teachers estimated that their student performance would be at the 
satisfactory or below satisfactory levels. Whereas, a higher percentage of students drought that their 
performance on MSPAP mathematics tasks was at the excellent level. In fact, only a very small 
percentage of students (5%) felt that their performance was below satisfactory. Although the results were 
similar across grades, a slightly higher percentage of 5 th grade students than 8 th grade students felt they 
did excellent (38% and 24%, respectively). 
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Beliefs About Meeting MSPAP Standards 

Approximately 56% of the principals somewhat or strongly agreed with the statement that the goal of 
meeting MSPAP standards by the year 2000 is a realistic goal for their school and only 20% strongly 
disagreed. Approximately 42% of the teachers somewhat or strongly agreed with this statement and only 
21% strongly disagreed. Further, when asked how important it was for them to see improved student 
performance on MSPAP from year to year, all the principals indicated that it was somewhat or very 
I important The majority of the teachers (85%) also indicated that it was somewhat or very important for 

them to see MSPAP performance gains from year to year. 

I 

Beliefs About Factors that Affect MSPAP Results 

| Principals and teachers were asked to indicate the extent to which various factors may contribute to 

MSPAP score gains for schools in the state. As shown in Table V.ll, both principals and teachers 
j indicated that there are a variety of factors that contribute a moderate or great extent to MSPAP score 

gains. The factors in the table are ordered according to the magnitude of the percent of principals who 
l reported that the factor contributes to a moderate or great extent to MSPAP score gains. 



Table V.l 1 Percentage of Principals and Teachers Indicatine the Factors that 
Affect MSPAP Score Gains in the State 



Factors 


Principals 


Teachers 


Increased student familiarity with the format of MSPAP 


97% 


88% 


i Use of MSPAP-like tasks in regular instruction 


97% 


86% 


Student improvement in knowledge and skills emphasized by 
MSPAP 


94% 


80% 


General improvement of instruction 


91% 


76% 


Use of MSPAP public release tasks or other performance 
tasks as test preparation materials 


86% 


79% 


Differences between student populations from year to year 


84% 


81% 


Improved test-taking skills 


82% 


79% 


Increased student motivation 


64% 


56% 



The two factors that received the highest percentages from both principals and teachers are increased 
j student familiarity with tire format of MSPAP and the use of MSPAP-like tasks in regular instruction. 

Similarly, the factor that received the smallest percentage from both principals and teachers is increased 
student motivation, although the percentages are still relatively high. 

The results were consistent across elementary and middle school principals for the factors 
i contributing to MSPAP score gains, with the exception of general improvement of instruction. Although 
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a similar percentage of elementary and middle school principals either somewhat or stron gl y agr eed that 
the general improvement of instruction contributes to score gains (93% and 85%, respectively), the level 
of agreement differs. About 52% of the elementary principals strongly agreed compared to only 21% of 
middle school principals. Teachers across grades responded similarly to the questions regarding factors 
that contribute to score gains. 
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Impact of MSPAP 



The results per tainin g to tile impact of MSPAP are provided in several sections: Overall impact of 
MSPAP, impact of MSPAP on instruction and classroom assessment, and impact of MSPAP on 
professional development activities. 

Overall Impact of MSPAP 

This section describes the overall impact of MSPAP on the principals, teachers and students as well 
as the impact of MSPAP on teacher and student morale. Principals and teachers were also asked to 
describe the positive and negative impacts. 



Overall Impact on Principals. Teachers, and Students . Principals were asked to indicate the 
extent of both positive and negative impact of MSPAP on themselves, their teachers, and their students. 
Teachers were asked to indicate the extent of both positive and negative impact on themselves and their 
students. Further, students were asked to indicate the extent of both positive and negative impact of 
MSPAP on their schools. 

Impact on Principals . The majority of the principals indicated that MSPAP had a moderate or very 
positive impact on them and that MSPAP had no or only a small amount of negative impact on them as 
summarized in Table V.12. When asked to respond to the statement that the positive impacts of MSPAP 
outweigh the negative impacts, 85% of the principals either somewhat or strongly agreed. The results 
were similar across elementary and middle school principals. 



Table V.12 Percenter nf Teachers and Principals Indicating the Extent of 
MSPAP’s Positive and Negative Impact 





Moderate or 


Small Amount 




Very Positive 


or No Negative 




Impact 


Impact 


Impact on Principal - 


82% 


75% 


Principal’s Perception 






Impact on Teacher 






Teacher’s Perception 


52% 


61% 


Principal’s Perception 


80% 


67% 


Impact on Student 






Teacher’s Perception 


37% 


66% 


Principal’s Perception 


74% 


86% 
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Principals were also asked to indicate the extent to which MSPAP had an impact on the time that 
they devote to instructional issues. Overall, 84% of the principals indicated that the time they devote to 
instructional issues has increased somewhat or greatly as a result of MSPAP. Elementary and middle 
school principals responded similarly to this question. 

Impact on Teachers . Approximately half of the teachers indicated that MSPAP had a moderate or 
very positive impact on them and a greater percentage of teachers indicated that MSPAP had no or only a 
small amount of negative impact on them. Teachers responded similarly across grade levels. Principals 
were also asked to indicate the extent to which MSPAP had a positive and negative infl uence on the 
teachers in their schools. Interestingly, 80% of the principals indicated that MSPAP had a moderate or 
very positive impact on teachers. These results indicate that principals, as compared to teachers, 
perceive that MSPAP has had a greater positive impact on teachers. Approximately, 67% of the 
principals indicated that MSPAP had no or only a small amount of negative impact on teachers. This 
result is similar to what the teachers indicated. 

When teachers were asked to respond to the statement that the positive impacts of MSPAP outweigh 
the negative impacts, 61% either somewhat or strongly agreed. This percentage is somewhat lower than 
the 85% of the principals who somewhat or strongly agreed with the statement 

Impact on Students . Less than half of the teachers reported that MSPAP had a moderate or very 
positive impact on students. However, 66% of the teachers indicated that MSPAP had no or only a small 
amount of negative impact on students. In contrast to the teacher results, 74% of the principals reported 
that MSPAP had a moderate or very positive impact on students and 86% reported that MSPAP had no 
or only a small amount of negative impact on students. 

Students were asked how much of a positive and negative impact MSPAP has had on their school. 
Overall, the results across grades were somewhat similar. About 28% to 40% of students in grades 4, 5, 
7, and 8 indicated that they did not know if MSPAP had a positive or a negative effect The 4 th grade 
students chose “don’t know” more often than the other grades and 8th grade students chose “don’t know” 
less often than the other grades (40% and 28%, respectively). A slightly higher percentage of 5th grade 
students (53%) compared to the other grade levels (about 42%) indicated that MSPAP has had a positive 
effect on their school. Whereas, more 8th grade students (31%), as compared to the other grade levels 
(about 20%), indicated that MSPAP has had a negative effect on their school. 

Comments from Principals and Teachers on the Impacts of MSPAP. In addition to indicating the 
extent of positive and negative impacts of MSPAP, the principals and teachers were also to explain these 
impacts and indicate their overall reaction to MSPAP. Of the 515 mathematics teachers who completed 
the questionnaire, 383 (74%) of the teachers provided comments explaining the impacts of MSPAP. 



Half of these teachers (50%) indicated that MSPAP had at least one positive impact and at least one 
negative impact, 39% of the teachers indicated only negative impacts, and 8% of the teachers indicated 
only positive impacts. The re maining 3% of teachers provided comments that were too vague to 
interpret or the teachers indicated that they were new and did not know about MSPAP. Of die 86 
principals who completed the questionnaire, 67 (78%) also provided comments about die impacts of 
MSPAP. A larger percentage of principals compared to teachers indicated at least one positive and at 
least one negative impact of MSPAP (72%). Only 13% of the principals indicated only negative impacts 
and 12% indicated only positive impacts. The remaining 3% of principals provided comments that were 
too vague to be interpreted. 

On average, a teacher’s response contained approximately 4 different types of positive and/or 
negative comments, while a principal’s response contained approximately 5 different comments. Thus, 
there were 1,536 comments coded for die teachers and 326 comments coded for principals. Table V.13 
below includes a list of several areas in which die positive and negative comments were classified. The 
percentages in the table reflect the percent of comments coded in each category. For example, 40% of all 
the positive comments provided by teachers (182 out of 454 comments) were related to the positive 
impacts MSPAP has had on curriculum, instruction, and student learning in die classroom. Similarly, 
41% of all the positive comments provided by principals (63 out of 152 comments) were related to 
positive impacts on curriculum, instruction, and student learning. 



Table V.13 Positive and Negative Comments About MSPAP from Teachers and Principals 



Positive Comments 


Percentage of 


Percentage of 




Teachers’ 


Principals’ 




Comments 


Comments 


Impact on Curriculum/Instruction/Student Learning 


40% 


41% 


Aspects of MSPAP 


37% 


29% 


Impact on Accountability 


6 % 


9% 


Professional Development Support 


3 % 


1% 


Impact on Attitudes and Morale 






Teachers 


5% 


6% 


Students 


4% 


4% 


Principal 


0% 


0% 


School 


4% 


7% 


Parents 


2% 


2% 








Negative Comment*; 


Percentage of 


Percentage of 




Teachers’ 


Principals’ 




Comments 


Comments 


Impact on Curriculum/Instruction/Student Learning 


11% 


5% 


Aspects of MSPAP 


30% 


29% 


Impact on Accountability 


23% 


36% 


Professional Development Support 


7% 


5% 


Impact on Attitudes and Morale 






Teachers 


13% 


12% 


Students 


12% 


5% 


Principal 


1% 


4% 


School 


2% 


1% 


Parents 


1% 


2% 



When comparing the percentages of positive and negative comments in each category, some 
interesting differences are found. With regard to die positive comments, the curriculum category 
contained the largest percent for teachers and principals. About half of the comments coded in this 
category indicated that there was an overall improvement of curriculum, instruction, assessment, or 
student learning because of MSPAP. The other half of the comments were related to more specific 
aspects of the curriculum, instruction, and assessment that improved such as more emphasis on higher- 
level thinking skills (e.g., problem solving and reasoning); teaching for conceptual understanding and 
making connections rather than procedural or rote memorization; more emphasis on mathematical 
communication; increase in real-world applications, use of MSPAP-like tasks/activities in instruction 
and assessment; and integration across subject areas. In contrast, for the negative comments, there was a 
smaller percentage of comments related to curriculum (11% for teachers’ comments and 5% for 
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principals’ comments). The most frequent types of negative impact on curriculum mentioned by 
teachers and principals were decreasing emphasis on basic skills (e.g., students not learning the 
foundational skills”), too much emphasis is placed on test preparation (e.g., “teaching to the test”), not 
enough tim e during the school year to use lengthy performance tasks, school/county curriculum does not 
coincide with the Maryland Learning Outcomes, and restricts creativity (e.g., “has taken the fun out of 
teaching and learning”). 

Another difference between the positive and negative comments is with respect to MSPAP’s impact 
on accountability. Only 6 % of the teachers’ positive comments and 9% of the principals’ positive 
comments were related to positive impacts on accountability such as raising expectations and standards 
for students, providing teachers with standards and goals to teach toward, and providing useful feedback 
for making improvements and m aintainin g successful activities. A larger percentage of negative 
comments were related to accountability (23% for teachers and 36% for principals). These comments 
included the pressure to improve scores, competition within schools and across schools because of 
publication of results, overly concerned about school’s test results and reputation and lack of concern 
about actual student learning, unrealistic expectations and goals, group scores rather than individual 
scores are focused upon, not appropriate to compare results for different student populations across years 
and across schools, and monetary rewards and reconstitution create tension. 

Approximately similar percentages of the positive and negative comments were related to specific 
aspects of MSPAP itself, for example 37% of the teachers’ positive comments and 30% of their negative 
comments were coded in this category. The most frequent positive comment about MSPAP was a belief 
in the overall goals and philosophy of MSPAP and that, overall, it is a positive way to assess students. 
Other specific comments were that the items are challenging, interesting, and engage students in higher- 
level thinking skills; items provide applications to die real-world, and the test is a good measure of 
students’ abilities in the content area. The most frequent negative comments about MSPAP include the 
developmental inappropriateness for average students and below-average students, the week of 
administration is too long for students and requires too much preparation and time away from 
ins truction, the items involve too much reading and/or writing, and the test does not accurately measure 
what students know in a content area. 

Only a small percentage of both the positive and negative comments were related to professional 
development activities. Positive comments in this category included the availability of good materials 
and resources from the county or school, more workshops and inservices, encouraged better use of 
planning time, meetings became more focused on instruction and teaching strategies, and encouraged all 
teachers (on-grade and off-grade) to collaborate with each other. Negative comments related to 
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professional development were the lack of appropriate materials, resources, support, and assistance; the 
lack of time to plan lessons, the lack of direct classroom-based support to apply what was learned in 
inservices. 

Finally, there were slightly larger percentages of negative comments compared to positive comments 
related to the attitudes and morale of teachers, students, principals, school, and parents. Examples of 
positive comments include “helped me to become a better teacher”, “as teachers become familiar with 
MSPAP, morale improves”, “students like die challenges in instruction”, “students are more aware of the 
purpose of learning and value it”, “students are proud of the schools’ ranking”, “school has a unifi ed 
focus and everyone works together toward a common goal”, and “parents are proud of the schools’ 
ranking” . The majority of negative comments were feelings of frustration and stress and that it is 
overwhelming to teachers and students. Other negative comments were “ more work for teachers”, 
“some teachers are resistant to change”, “students are not motivated because no individual scores are 
provided”, “school is obsessed with MSPAP/too much emphasis”, and “parents are concerned about 
MSPAP”. 

Impact on Teacher and Student Morale . The majority of principals indicated that MSPAP had a 
somewhat or very positive impact on teacher morale (53%) and student morale (64%) in their schools, 
with less than half of the principals indicating that MSPAP had a somewhat negative impact on teacher 
morale (41%) and student morale (32%). However, the majority of teachers reported that MSPAP had a 
somewhat negative impact on teacher morale (62%) and on student morale (63%). Despite such reports 
of negative impacts, 61% of the teachers either somewhat or strongly agreed with the statement that the 
positive impacts of MSPAP outweigh the negative impacts. 

The elementary and middle school principals reported similar levels of impact on teacher and student 
morale, and teachers across grade levels reported similar levels of impact on teacher and student morale. 

Impact of MSPAP on Instruction and Classroom Assessment 

This section describes the a) overall impact of MSPAP, the MLO’s and the Maryland Curriculum 
Framework; b) overall impact of MSPAP on instruction and assessment; c) overall impact of MSPAP on 
procedural skills, problem solving, and explanations; d) impact of MSPAP on learning outcomes; e) 
impact of MSPAP on the use of MSPAP-like tasks; f) impact of MSPAP on teacher- and student- 
centered classroom activities; g) impact of MSPAP on instruction task types; h) impact of MSPAP on 
assessment task types; i) impact of various features of MSPAP; and j) preparing students for MSPAP. 
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For some of the questions regarding classroom activities and task types, teachers and principals were 
first asked to indica te the occurrence of certain activities in the 1996-97 instruction year, and they were 
then asked to indicate die extent to which the frequency of these activities changed from 1992-93 (the 
initial years of MSPAP). Only teachers who taught in a Maryland school during the 1992-93 year 
responded to the questions pertaining to the change in emphasis. Consequently, approximately 70% of 
the teachers responded to these questions. Students were only asked to indicate the extent to which the 
classroom activities occurred during the 1996-97 instruction year. 

Overall Impact of MSPAP. MLO’s. and Maryland Curriculum Framework. Teachers were 
asked to indicate the extent to which MSPAP, the Maryland Learning Outcomes (MLO’s), and die 
Maryland Curriculum Framework (MCF) had an impact on their classroom activities and to indicate the 
extent to which the impact changed since 1992-93. The table below summarizes the findings. The values 
in the Table V .14 ind ic ate the percentage of teachers who responded in either the “moderate” or “great ” 
impact categories. 



Table V.14 Percentage of Teachers Indicating that Various Sources had a 
Moderate or Great Amount of Impact 



Source of 
Impact 


Moderate or Great 
Impact 
1996-97 


Moderate or Great 
Increase in Impact 
1992-97 


MSPAP 


82% 


83% 


MLO’s 


72% 


56% 


MCF 


64% 


44% 



Overall, the majority of the teachers indicated that MSPAP had a moderate or great influence on their 
classroom activities, and tire majority indicated that the influence increased a moderate or great extent 
since 1992-93. The majority of the teachers also indicated that the MLO’s and Maryland Curriculum 
Framework had a moderate or great impact, but to a lesser extent than MSPAP s impact. 

There were several differences across grade levels as summarized in Table V.15 below. In particular, 
there was a larger percentage of 3^ and 5^ grade teachers as compared to the other grade-level teachers 
who indicated that both MSPAP and the MLO’s had a “great” impact and that the impact increased a 
“great” extent since the inception of MSPAP (1992-93). Whereas, the 7 th grade teachers had the 
smallest percentages in the “great” impact category. 
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Table V.15 Percentage of Teachers Indicating that Various 
Sources had a Great Amount of Impact 



Source of 
Impact 


3 rd and 5^ 


2 nd and 4 th 


8 th 


7 m 


Great 

Impact 

1996-97 


Great 

Increase 

1992-97 


Great 

Impact 

1996-97 


Great 

Increase 

1992-97 


Great 

Impact 

1996-97 


Great 

Increase 

1992-97 


Great 

Impact 

1996-97 


Great 

Increase 

1992-97 


MSPAP 


58% 


51% 


32% 


35% 


40% 


34% 


21 % 


25% 


MLO’s 


42% 


34% 


34% 


26% 


30% 


16% 


11 % 


12 % 



Overall Impact of MSPAP on Instruction and Assessment . Teachers and principals were asked to 
indicate the extent to which MSPAP influenced them (their teachers) to make positive changes in their 
(a) instruction and (b) assessment. Overall, die majority of the teachers indicated that MSPAP had 
either a moderate or great amount of positive impact on their instruction (76%) and assessment ( 68 %). 
These percentages are larger for the results shown earlier regarding the overall positive impact of 
MSP AP , in that, only 52% of the teachers indicated that MSPAP had an overall positive impact on them 
and only 37% of the teachers indicated that MSPAP had an overall positive impact on their students. 
However, the results reported in this section indicate that when asked specifically about the impact of 
MSPAP on instruction and assessment, a relatively large percentage of teachers indicated that MSPAP 
had a moderate or great amount of positive impact Table V.16 provides the percentages of teachers and 
principals by grade level who indicated that MSPAP had either a moderate or great am ount of positive 
impact on instruction and assessment for the elementary and middle school on- and off-grade levels. 
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Table V.16 Percentage of Teachers and Principals Indicating that MSPAP had a 
Moderate or Great Amount of Positive Impact on Instruction and Assessment 



Grade Level 


Teachers 

Moderate/Great Amount of 
Positive Impact 


Principals 

Moderate/Great Amount of 
Positive Impact 


Instruction 


Assessment 


Instruction 


Assessment 


3 1X1 and 5“ 


83% 


75% 


96% 


93% 


2nd and 4^ 


76% 


70% 


89% 


85% 


8th 


74% 


59% 


96% 


85% 


7 tn 


64% 


57% 


86% 


78% 



As shown in the table, 3°* and 5*h grade teachers more often than the other teachers indicated that 
MSPAP had a moderate or great amount of positive impact on instruction and assessment In contrast to 
the teachers’ results, the middle school principals reported a similar level of impact for 8^ grade as the 
elementary principals reported for the 3*^ and 5^ grades. However, for the question about MSPAP’s 
positive impact on instruction, elementary school principals chose the category “a great amount” for 3*^ 
and 5 th grade teachers 73% of the time, whereas middle school principals chose this category for 8 th 
grade teachers only 37% of the time. 

Further, when asked to respond to the statement that the state has defined curriculum content well 
enough to make instruction and assessment changes so as to reflect the goals of MSPAP, die majority of 
the principals (62%) and the teachers (52%) somewhat or strongly agreed. 

Overall Impact of MSPAP on Procedural Skills, Problem Solving, and Explanations . Teachers 
and students were asked to indicate the extent to which mathematical procedures and factual knowledge, 
problem solving and reasoning, and mathematical explanations and justifications were emphasized in 
their classroom during 1996-97. Further, teachers and principals were asked to indicate the extent to 
which the emphasis on these areas changed since the beginning of MSPAP (1992-93). 

As indicated in Table V.17, the majority of teachers reported that they emphasized mathematical 
procedures and factual knowledge, problem solving and reasoning, and mathematical justifications and 
explanations either daily or at least once a week. The students’ reported that their classrooms’ 
emphasized the three areas somewhat less than what the teachers reported. For example, teachers, as 
compared to students, indicated that problem solving and reasoning was emphasized to a greater extent 



Table V.17 Percentage of Teachers and Students Indicatine Classroom Coverage of 
Procedures. Problem Solving & Explanations 





Teachers 


Students 


Daily/Weekly 

1996-1997 


Moderate/Great 

Increase 

1992-1997 


Daily/Weekly 

1996-1997 


Procedure s/F actual 
Knowledge 


88 % 


42% 


73% 


Problem Solving/ 
Reasoning 


82% 


75% 


54% 


Explanation/ 

Justification 


68 % 


81% 


49% 



Approximately 42% of the teachers indicated that the emphasis on mathematical procedures and 
factual knowledge increased a “moderate” or “great” extent since 1992-93, with 43 % of the teachers 
indicating that the emphasis was the same. However, the majority of die teachers indicated that the 
emphasis on problem solving and reasoning and mathematical explanations and justifications increased a 
“moderate” or “great” extent It appears that teachers have placed a much greater emphasis on students’ 
problem solving, reasoning, and use of explanations since the inception of MSPAP. 

There were several differences across the grade levels for the teachers. A larger percentage of 
elementary on-grade (3°* and 5 *h) teachers reported dial on a daily or weekly basis they emphasized 
problem solving and reasoning (89%) and the use of explanations (82%) as compared to off-grade ( 2 n d 
and 4*) elementary teachers (83% and 68 %, respectively), on-grade ( 8 th) middle school teachers ( 74 % 
and 50%, respectively), and off-grade ( 7 *) middle school teachers (67% and 50%, respectively). The 
differences across grades is primarily in terms of die use of explanations. The change in emphasis was 
similar across grades. 

Student responses to the questions regarding the emphasis of mathematical procedures and problem 
solving and reasoning were similar across grade levels. However, similar to the teacher results, there 
were differences between the elementary and middle grades with respect to the emphasis on 
mathematical justifications and explanations. About 30% of 4 th ^ 5 th grade students said they work 
daily on problems asking them to explain in writing how they got their answer, while only 14% of the 
7 th and 8 th grade students indicated that this activity occurred daily. A larger percentage of die middle 
grade students (24%) said they never or only 3 to 4 times a year were asked to explain their answers 
compared to 1 0 % of the elementary students. 

Principals were asked to indicate how the emphasis of instruction, with respect to mathematical 
procedures and factual knowledge as well as mathematical problem solving and reasoning, changed from 
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1992*93 in the teachers’ classrooms. The majority of the elementary principals indicated that the on- 
grade (3^ and 5^) and off-grade (2°d and 4&) teachers’ emphasis on mathematical procedures and 
factual knowledge increased somewhat or greatly since 1992-93, 68% and 66%, respectively. Further, a 
larger percentage of elementary principals indicated that the on-grade (3*^ and 5th) and off-grade (2 n< * 
and 4th) teachers’ emphasis on mathematical problem solving and reasoning increased somewhat or 
greatly, 86% and 81%, respectively. The results from the middle school principals followed a pattern 
similar to the elementary school principals’ results. The majority of die middle school principals 
indicated that the on-grade (8 th ) and off-grade (7 th ) teachers’ emphasis on mathematical procedures and 
factual knowledge increased somewhat or greatly since 1992-93, 63% and 50%, respectively. Further, a 
larger percentage of elementary principals indicated that the on-grade (8*) and off-grade (7™) teachers’ 
emphasis on mathematical problem solving and reasoning increased somewhat or greatly, 80% and 75%, 
respectively. 

Thus, the principals’ results support the teachers’ results indicating that problem solving and 
reasoning has become a much greater focus in the classrooms since the beginning of MSPAP. 

Impact of MSPAP on Learning Outcomes . Teachers and students were also asked a set of 
questions pertaining to the focus of their classroom activities with respect to MSPAP’ s process outcomes 
and content outcomes. 

MSPAP’ s Process Outcomes . Most of the teachers indicated that they emphasize the process 
lea rning outcomes to a moderate or great extent and the increase in emphasis since the beginning of 
MSPAP has increased moderately or greatly as summarized in Table V.18. 

Table V.18 Percentage of Teachers Indicating Emphasis and Increase in 
Emphasis of MSPAP’s Process Outcomes 



Process Outcomes 


Moderate or 
Great Emphasis 
1996-97 


Moderate or 
Great Increase 
in Emphasis 
1992-97 


Problem Solving 


96% 


72% 


Communication 


87% 1 


74% 


Reasoning 


93% 


69% 


Connections 


87% 


65% 



The teachers across grades indicated a similar emphasis on problem solving, communication, 
reasonin g, and connections; however, there were differences for the change in emphasis on problem 



solving and reasoning. Although the majority of the teachers across grades (63% to 77% depending on 
grade) indicated that die emphasis on problem solving increased somewhat or greatly, the extent of 
increase differed. Overall, a larger percentage of elementary on-grade (3 rd and 5^) teachers indicated 
that the increase in problem solving was great (33%) as compared to the other grade levels (10% to 15% 
depending on grade). The greatest difference across grades in the reported change in emphasis on 
reasoning was between die elementary on-grade (3 rd and 5^) teachers and the middle school on-grade 
(8*h) teachers. Approximately 77% of die elementary on-grade teachers indicated that change in 
emphasis on reasoning increased somewhat or greatly as compared to 56% of the middle-school on- 
grade teachers. 

MSPAP > s Content Outcomes . The majority of the teachers also indicated that they emphasize the 
content learning outcomes a moderate amount or a great deal as summarized in Table V.19 below. 
Teachers tended to indicate that the emphasis on these content outcomes in instruction either stayed the 
same or increased somewhat or greatly since 1992-93. Overall, it appears that none of the learning 
outcome emphases decreased over time. 



Table V.19 Percentaee of Teachers Indicating Emphasis and Increase in 
Emphasis of MSPAP’s Content Outcomes 



Content Outcomes 


1996-1997 


1992-1997 


Moderate or Great 
Emphasis 


Moderate or Great 
Increase in Emphasis 


Same Amount of 
Emphasis 


Numbers and Operations 


94% 


50% 


48% 


Geometry/Measurement 


84% 


51% 


46% 


Statistics 


58% 


54% 


42% 


Probability 


53% 


54% 


42% 


Pattems/Algebra 


74% 


60% 


39% 



There were several differences across grade levels in terms of the extent to which the teachers 
emphasized the content outcomes in the 1996-97 instructional year. Elementary grade teachers reported 
a greater emphasis on geometry/measurement as compared to middle school teachers, with 89% of the 
2 nd and 4*h grade teachers and 92% of the 3 rd and 5^ grade teachers, as compared to 78% of the 7*h 
grade teachers and only 63% of the 8^ grade teachers, indicating that they emphasize 
geometry/measurement to a moderate or great extent The 2 nd and 4 th grade teachers indicated less 
emphasis on statistics than the other grade level teachers. Only 46% of the 2 nd and 4^ grade teachers, 
as compared to 65% of the 3 rd and 5^ grade teachers, 70% of the 7^ grade teachers and 55% of the 8^ 
grade teachers, indicated that they emphasize statistics to a moderate or great extent. The 3 rd and 5th 
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grade teachers reported a greater emphasis on probability as compared to the other grade-level teachers, 
in that, 65% of the 3 rc * and 5^ grade teachers indicated that they emphasize probability to a moderate or 
great extent, while only 48% of the 2 n< * and grade teachers, 48% of the 7^ grade teachers, and 43% 
of the 8th grade teachers so indicated. Lastly, although there were similar percentages of teachers across 
grade levels indic atin g that they emphasized patterns/ algebra to a moderate or great extent (68% to 
79%), a larger percentage of 8 th grade teachers (47%) indicated that they emphasized patterns/algebra to 
a great extent as compared to the other grade teachers (24% to 35%). The heavy emphasis on 
pattems/algebra in the 8^ grade may help explain why their emphasis on other content areas was less as 
compared to the other grade levels. 

Impact of MSPAP nn Use of MSPAP-like Tasks in the Classroom . Teachers were asked to 
indicate how often they provide students with the opportunity to solve tasks similar to those on MSPAP. 
Similarly, students were asked to indicate the extent to which they have the opportunity to solve 
MSPAP-like tasks. Overall, 31% of the teachers indicated that students have the opportunity weekly or 
daily, 47% indicated every two weeks or monthly, 21% indicated about 4 times per year, and 2% said 
never. Overall, 38% of die students indicated that they solve such tasks weekly or daily, 33% indicated 
every two weeks or monthly, 22% indicated about 4 times per year, and 7% said never. In general, the 
results for teachers and students are similar, except that middle school students tended to indicate a 
greater use of MSPAP-like tasks than did their teachers. 

As indicated in Table V.20, on-grade elementary teachers (3°* and 5^) indicated that they use tasks 
similar to those on MSPAP most frequently and 7*h grade teachers indicated that they use them the least 
frequently. For example, 43% of die 3^ and 5*h grade teachers indicated that they use them on a daily 
basis, whereas 14% of the 7^ grade teachers indicated that they use them on a daily or weekly basis. 
Similarly, 43% of die 7& grade teachers indicated that they use them only about 4 times per year, 
whereas only 10% of the 3^ and 5^ grade teachers indicated that they use them about 4 times per year. 

Overall, the student results across grades are similar. However, there was a slight difference between 
off- and on-grades in that a smaller percentage of 4*h and 7^ grade students (35% and 31%, 
respectively) reported that they worked on MSPAP-like tasks daily or weekly when compared to the 5^ 
grade (44%) and 8^ grade students (40%). 
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Table V.20 Percentage of Teachers and Students Indicating Use of 
MSPAP-like Tasks in the Classroom 



Teachers 


Daily or 
Weekly 


Every 2 Weeks/ 
Monthly 


3 or 4 times 
per year 


Never 


All teachers 


31% 


47% 


21 % 


2 % 


3rd and 5 m 


43% 


47% 


10 % 


0 % 


2 nd and 4® 


27% 


50% 


24% 


0 % 


— 8 ® 


26% 


47% 


24% 


3% 


7® 


14% 


38% 


43% 


5% 




Students 


Daily or 
Weekly 


Every 2 Weeks/ 
Monthly 


3 or 4 times 
per year 


Never 


All students 


38% 


33% 


22 % 


7% 


5 tn 


^ 44% 


34% 


17% 


5% 


4 th 


35% 


34% 


24% 


7% 


8 tn 


40% 


32% 


20 % 


8 % 


7 th 


31% 


32% 


31% 


6 % 



Teachers were also asked about their use of MSPAP public release tasks and Performance-Based 
School Improvement Exemplars for classroom purposes. Approximately 41% of the teachers indi c-a ft-d 
that they used the MSPAP public release tasks about 4 times a year and 1 8% indicated that they use them 
monthly, whereas 33% indicated that they never use MSPAP public release tasks for classroom purposes. 
Further, 35% of the teachers indicated that they used the Performance-Based School Improvement 
Exemplars about 4 times per year and 16% indicated that they use them monthly. Whereas, 39 % 
indicated that they never use Performance-Based School Improvement Exemplars for classroom 
purposes. 

On-grade elementary teachers (3^ and 5 *h) were most likely to use MSPAP public release tasks and 
Performance-Based School Improvement Exemplars for classroom purposes and 7^ grade teachers were 
least likely to use them. For instance, 75% of the 5 th grade teachers indicated they use MSPAP public 
release tasks monthly or about 4 times per year and only between 43% and 53% of the other grade level 
teachers indicated that they use them monthly or about 4 times per year. Further, only 13% of the 5 ^ 
grade teachers indicated they never use MSPAP public release tasks, whereas between 41% and 53 % of 
the other grade teachers indicated that they never use MSPAP public release tasks. 

Impact of MSPAP on Teacher- and Student-Centered Classroom Activities . Teachers and 
students were asked to indicate the extent to which classroom activities and discussions were teacher- or 
student-centered. Overall, 77% of the teachers indicated that they presented to the class daily, 60% 
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indicated whole class discussions occurred daily, and 63% indicated that individual student seat work 
occurred daily. A smaller percentage of teachers (42%) indicated that student group work occurred daily; 
however, an additional 44% of die teachers indicated that group work occurred at least once a week. The 
majority of the teachers indicated that the emphasis on teacher presentations (70%), whole class 
discussions (61%), and individual seat work (60%) remained the same since 1992-93. Whereas, 42% of 
the teachers indicated that group work increased somewhat since 1992-93 and 37% indicated that the 
emphasis on group work remained the same. 

There were two general differences across grade levels in the type of classroom activities reported. 
Elementary school teachers reported more frequent use of whole class discussions than middle school 
teachers. Secondly, elementary school teachers reported more frequent use of small group work, with 
nearly 50% of he elementary teachers indicating that small group work occurs daily and only 30% of 
middle school teachers indicating that small group work occurs daily. 

Overall, 81% of he students indicated that teachers presented to he class daily, 42% of he students 
indicated hat whole class discussions occurred daily, and 60% indicated he hey did individual seat 
work daily. These results are similar to he results reported by he teachers. However, he percentage of 
students that indicated that hey worked in groups either daily or weekly was lower than what he 
teachers’ reported, 10% and 22%, respectively. 

Wih respect to teachers presenting to he class, whole class discussions, and individual seat work, 
he percentages across grade levels for students were very similar. For small group work, however, 
larger percentages of off-grade students (4^ and 7^) said hey never or only 3 to 4 times a year worked 
in small groups (34%) while only 18% of he 5*h grade students and 25% of he 8*h grade students 
selected hese categories. 



Impact of MSPAP on Instruction Task Types 

Instruction Task Types . Teachers and students were asked to indicate he use of various task types in 
heir mathematics classrooms. Teachers were also asked to indicate how he emphasis on task types 
changed since he beginning of MSPAP (1992-93). 

Although teachers reported using textbook problems more frequently than open-ended problems, he 
majority of he teachers indicated that since 1992-93 he use of open-ended problems increased 
somewhat or to a great extent as indicated in Table V.21 below. Whereas, most of he teachers indicated 
hat he use of textbook problems stayed he same or decreased. Teachers reported less frequent use of 
extended problems that take a few days to complete, which is not unreasonable given he amount of time 
hat is required to solve such tasks, but overall, 57% of he teachers indicated hat he use of extended 
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problems increased moderately or greatly since 1992-93. The majority of teachers indicated that they 
use tasks that apply mathematics to real-life situations daily or weekly, and the use of such tasks has 
increased since 1992-93. The majority of teachers also indicated that they use tasks that require 
manipulatives daily or weekly, and die use of manipulatives has increased. Further, half of the teachers 
indicated that they use tasks that integrate other subject areas in math on a daily or weekly basis, and the 
use of integrated tasks has increased. 

In contrast to the teachers’ results, the percentage of students who indicated that they worked on 
open-ended problems (i.e., problems that ask you to show your work and explain your answer) and 
textbook problems daily or weekly was similar, 77% and 73%, respectively. The students’ reported use 
of open-ended problems (73%) was greater than the teachers’ reported use (59%). As did the teachers, 
however, students indicated that they less frequently worked on extended problems that take a few days 
to complete, with only 1 8% of the students indicating that they work on extended problems daily or 
weekly. However, a larger percentage of students (31%) than teachers (17%) indicated that extended 
problems are never used in instruction. Further, a smaller percentage of students than teachers indicated 
that they solve tasks that apply mathematics to real life situations, use manipulatives, and integrate other 
subject areas in math daily or weekly. 



Table V.21 Percentage of Teachers and Students Indicating Emphasis and Impact of MSPAP 

On the Use of Instruction Task Types 



Task Types 


Emphasis 

1996-97 


Change in Emphasis 
1992-97 


Teachers 


Daily/ 

Weekly 


Biweekly/ 

Monthly 


4 times 
per year 


Never 


Increase 


Same 


Textbook 


74% 


18% 


4% 


4% 


11% 


59% 


Open-ended 


59% 


32% 


6% 


3% 


65% 


33% 


Extended 


18% 


41% 


24% 


17% 


57% 


39% 


Real-life 


78% 


20% 


1% 


0% 


66% 


32% 


Manipulatives 


59% 


34% 


7% 


1% 


52% 


44% 


Integrated 


50% 


40% 


9% 


1% 


64% 


34% 


Students 














Textbook 


73% 


| 17% 


6% 


4% 


— 


— 


Open-ended 


77% 


17% 


4% 


1% 


— 


— 


Extended 


18% 


31% 


21% 


31% 


— 


— 


Real-life 


46% 


28% 


14% 


13% 


— 


— 


Manipulatives 


30% 


39% 


21% 


11% 


— 


— 


Integrated 


26% 


39% 


22% 


13% 




— 
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Differences Across Grade Levels on Instruction Task Types . There were several differences 
between elementary and middle school teachers as well as between on- and off-grade teachers. In 
general, elementary teachers as compared to middle-school teachers indicated a more frequent use of 
open-ended problems, with 3 rd and 5 *h grade teachers reporting the greatest increase in use. Whereas, 
middle-school teachers as compared to elementary school teachers indicated a more frequent use of 
textbook problems, with the elementary school teachers indicating a greater decrease in use. Further, 
elementary teachers tend to use integrated math tasks more frequently than middle school teachers. 

With regard to on- and off-grade teachers, approximately 50% of die off-grade (2 nd , 4^, 7*h) 
teachers indicated that they never use extended tasks or only use diem about 4 times per year, whereas 
only 30% 3 rd and 5th grade teachers and 40% of the 8 th grade teachers indicated that they never use 
extended tasks or only use them about 4 times a year. Further, 3 rd and 5 th grade teachers tended to show 
a greater increase in use of extended tasks since the inception of MSPAP. 

There were also differences for die student results across grades. As the grade-level increased, the 
percentage of students who indicated that they have textbook problems weekly or daily increased. About 
61% of 4 th grade students reported frequent use compared to 84% of 8 th grade students. A smaller 
percentage of students in 4 th grade (69%) versus students in 5 th grade (83%) indicated that they worked 
on open-ended problems either weekly or daily. The percentages for die other two grade levels were 
similar to the 5 th grade reported percentage. No differences were found across grades for frequency of 
working on extended problems. With respect to manipuladves, the elementary grades ( 4 th and 5 th) 
indicated more frequent use than the middle grades, at least every 2 weeks. The difference was 
especially noticeable between the 5 th grade (56%) and the 7 th grade (34%). About 50% of the 5 th grade 
students indicated that they work on problems integrating other subjects in math at least every 2 weeks. 
This percentage is slighdy more than the other grades, especially in 7 th grade where only 35% of the 
students reported this level of use. Lastly, there were no differences across the grade-levels with regard 
to students’ reported use of math to solve real-life problems. 

Classroom Instruction’s Impact on MSPAP Performance . Students in both 5 th and 8 th grade were 
also asked to indicate the extent to which the problems and activities in their math classes helped them 
answer the math tasks on MSPAP, and 4 th and 7 th grade students were asked to indicate the extent to 
which their math classes would help them answer tasks like those on MSPAP. The 4 th and 7 th grade 
students were given a released MSPAP task to examine prior to completing the questionnaire. Overall, 
80% of the students indicated that their classroom activities helped them (or would help them) answer 
the math tasks on MSPAP a moderate amount or very much. Whereas, the percentage of students who 



60 






O 



61 



indicated that their classroom activities did not help them (or would not help them) at all to solve 
MSPAP tasks was only 3%. 

The differences between the percentage of students who indicated that the problems in their math 
classes helped them answer the MSPAP tasks a moderate amount or very much were not large. The 
largest difference being between die on-grade elementary (3^ and 5^) students and die on-grade middle 
school (8*) students, 88% and 72%, respectively. However, 53% of the 3°^ and 5th grade students chose 
the category “very much” and only 32% of the 8th grade students chose this category. 

Impact of MSPAP on Assessment Task Types . Teachers and students were asked to indicate the 
extent to which various task formats are used for classroom assessments. Teachers were asked to 
indicate the extent to which they use various assessment task types and how the emphasis has changed 
since the beginning of MSPAP, whereas students were only asked to indicate the extent to which the 
various task types are used in their classrooms. 

Traditional Assessment Task Types . In general, teachers reported that they use short answer items 
more often than multiple-choice and end-of-unit textbook items as shown in Table V.22. Further, 
teachers indicated that their use of short answer items has increased more than their use of multiple- 
choice and textbook items. Overall, the students’ results were similar to the teachers’ results with one 
exception; students, as compared to teachers, reported a much greater use of textbook items. Further, as 
the grade-level increased, so did the frequency with which they were used. For example, 49% of 4th 
grade students versus 69% of 8th grade students reported working on them every week. These results are 
similar to those reported in an earlier section regarding use of textbook problems in instruction 
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Table V.22 Percentage of Teachers and Students Indicating the Emphasis and Impact of 
MSPAP On the Use of Traditional Assessment Task Types 



Task Type 


Emphasis 

1996-97 


Change in Emphasis 
1992-97* 


Teacher 


Weekly 


Biweekly 

/ 

Monthly 


4 times 
per year 


Never 


Increase 


Same 


Multiple Choice 


9% 


38% 


30% 


23% 


13% 


53% 


End of unit text items 


9% 


54% 


19% 


17% 


13% 


68% 


Short Answer 


34% 


54% 


8% 


4% 


36% 


54% 


Students 


Multiple Choice 


13% 


47% 


29% 


12% 


— 


— 


End of unit text items 


60% 


24% 


10% 


6% 


— 




Short Answer 


32% 


44% 


17% 


6% 


— 





* The increase category combines “moderate” and “great” increase. 



Use of Alternative Assessment Task Types as Reported bv Teachers . As shown in the Table V.23 
below, the reported use of open-ended problems for assessments was more frequent titan the use of 
multiple-choice items, with 25% of the teachers indicating that they use open-ended tasks weekly and 
56% indicating that they use them biweekly or monthly. Further, 60% of the teachers indicated that their 
use of open-ended tasks increased to a moderate or great extent since the beginning of MSPAP. 
Compared to the use of open-ended tasks, the use of extended tasks that take a few days to complete 
occurs less often. However, this might be expected due to the time required for extended tasks. 
Approximately 8% of the teachers indicated that they use extended tasks weekly and 48% indicated that 
they use them bimonthly or monthly. More importantly, however, the majority of the teachers (52%) 
indicated that they have increased the use of such tasks on assessments to a moderate or great extent 
Approximately 88% of the teachers indicated that manipulatives were used with assessments weekly, 
bimonthly or monthly, and 50% indicated that the use of manipulatives increased to a moderate or great 
extent. 

The majority of the teachers (86%) indicated that they use assessments which require written 
explanations weekly, biweekly or monthly, and 73% of the teachers indicated that the requirement of 
written explanations increased somewhat or greatly. Of particular interest is that 40% of the teachers 
indicated that they ask students to write in their journals on a weekly basis. Another 27% of the teachers 
reported that they ask students to write in their journals bimonthly or monthly. Further, the majority of 
the teachers indicated that the use of journals increased somewhat or greatly since 1992-93. The reported 
increased use of journal writing and written explanations supports the previously reported increased 
focus on communication in the classrooms. 
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Table V.23 Percentaee of Teachers and Students Indicating the Emphasis and Impact of MSPAP 
On the Use of Alternative Assessment Task Types 



Task Type 


Emphasis 

1996-97 


Change in Emphasis 
1992-97 


Teacher 


Weekly 


Biweekly/ 

Monthly 


4 times 
per year 


Never 


Increase 


Same 


Open-ended 


25% 


56% 


15% 


6% 


60% 


38% 


Extended 


8% 


48% 


32% 


12% 


52% 


44% 


Manipulatives 


48% 


40% 


10% 


3% 


50% 


47% 


Explanations 


40% 


46% 


11% 


4% 


73% 


25% 


Journals 


40% 


27% 


12% 


21% 


54% 


42% 


Portfolios 


11% 


25% 


14% 


50% 


39% 


57% 


Group activity 


48% 


41% 


8% 


3% 


58% 


41% 


Scoring rubric 


22% 


45% 


23% 


11% 


72% 


25% 


Students 


Open-ended 


57% 


33% 


8% 


2% 


— 


— 


Extended 


3% 


36% 


42% 


19% 


— 


— 


Manipulatives 


13% 


44% 


32% 


11% 


— 


— 


Explanations 




— 


— 




— 


— 


Journals 


20% 


17% 


11% 


52% 


— 


— 


Portfolios 


6% 


15% 


14% 


64% 




— 


Group activity 


8% 


42% 


34% 


17% 


— 


— 


Scoring Rubric 




— 


— 


— 


— 


— 



In general, teachers indicated that they used portfolios less often than the other assessment formats; 
however, teachers indicated an increase in their use since 1992-93. Most of the teachers indicated that 
group assessments occur weekly, biweekly or monthly, and 58% of the teachers indicated that the use of 
group assessments has increased to a moderate or great extent 

Overall, 67% of the teachers reported that they use scoring rubrics weekly, biweekly or monthly. 
Only 1 1% of the teachers indicated that they do not use scoring rubrics. Moreover, 72% of the teachers 
indicated that the use of rubrics increased to a moderate or great extent A promising finding from these 
results is the increased use of open-ended tasks, extended tasks, journals, portfolios, group assessments, 
and scoring rubrics in mathematics classrooms. 

Use of Alternative Assessment Task Types as Reported by Students . As indicated in Table V.23 
above, students indicated a greater use of open-ended problems than did their teachers and they indicated 
about the same amount of use of extended tasks. However, students, as compared to their teachers, 
reported less frequent use of tasks that involve manipuladves, journals, portfolios, and group 
assessments. 
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Students were also asked whether their teachers explained to diem how student work was graded. 
Approximately, 80% of the students indicated “yes”. There was no difference across grade levels. This 
question also asked students to explain how their work was graded. 

Teacher Differences Across Grade Levels for Alternative Task Types . Differences across grade 
levels for teachers occurred for the use of open-ended problems, manipulatives, written explanations, 
journals, portfolios, and scoring rubrics. In general, elementary school teachers, as compared to middle 
school teachers, indicated more frequent use of these types of assessments. Elementary school teachers 
as compared to middle school teachers indicated a more frequent use of open-ended problems, with 3^ 
and 5*h grade teachers reporting the most frequent use (66% indicated weekly or biweekly (i.e., every 2 
weeks)) and 7^ grade teachers reporting the least frequent use (only 35% indicated weekly or biweekly). 
Further, 3 r d and 5*^ grade teacher reported the greatest change in frequency of use and the 7*h grade 
teachers reported the least amount of change in use. 

Elementary school teachers, as compared to middle school teachers, indicated a more frequent use of 
manipulatives with classroom assessments, with 65% of the 2 Q d/4 t h grade teachers and 55% of the 
3 td/ 5 th grade teachers indicating that they have student use manipulatives with classroom assessments 
weekly and only 14% of the 7^ grade teachers and 26% of the 8^ grade teachers indicating such 
frequent use of manipulatives. 

Approximately 77% of the 3^/5^ grade teachers and 66% of the 2^/4^ grade teachers require 
students to provide written explanations weekly or biweekly, whereas 54% of the 8*h grade teachers and 
only 43% of the 7*h grade teachers do so. Although the majority of the teachers in each of the grades 
indicated that the use of written explanations increased since the inception of MSPAP, the extent of the 
increase differed across grade levels. Forty-six percent of the 3 r ^/5^ grade teachers and 37% of the 
2 Q d/ 4 th grade teachers indicated that they increased the use of written explanations a “great” extent, 
whereas 21% of the 8^ grade teachers and 14% of the 7*h grade teachers indicated that they increased 
the use to a “great” extent Approximately 66% of the elementary school teachers and only 23% of die 
middle school teachers have students write in journals on a weekly or biweekly basis. Further, 64% of 
the 3 r d/5th grade teachers and 57% of the 2^/4^ grade teachers indicated that their use of journals 
increased somewhat or greatly since the inception of MSPAP, whereas 39% of the 8^ grade teachers and 
only 28% of the 7*h grade teachers indicated that their use of journals increased. 

Approximately 26% of the on- and off-grade elementary school teachers indicated that they use 
portfolios on a weekly or biweekly basis, whereas 16% of the 8*h grade teachers and only 6% of the 7^> 
grade teachers indicated weekly or biweekly use of portfolios. Further, the majority of the 8*h grade 
(65%) and the 7*h grade (70%) teachers indicated that they never use portfolios, whereas only 45% of the 



3 rd/ 5 th grade and 38% of the 2 n d/4th grade teachers indicated they never use them. Approximately 50% 
of die elementary school teachers indicated that their use of portfolios has increased, whereas less than 
25% of the middle school teachers have reported an increase use of portfolios. 

Teachers of 3^ and 5*h grades reported the most frequent use of scoring rubrics and 7* grade 
teachers reported they use them the least amount, with 54% of the 3*^/5^ grade teachers indicating that 
they use rubrics weekly or biweekly and only 19% of the 7^ grade teachers indicating that they use 
rubrics weekly or biweekly. In addition, 82% of the 3*^/5^ grade teachers indicated that their use of 
rubrics has increased somewhat or greatly and between 63% and 68% of die other grade levels indicated 
that their use of rubrics has increased. 

Student Differences Across Grade Levels for Alternative Task Types . Differences across elementary 
and middle school grades occurred for the use of journals and portfolios. About 74% of middle school 
students compared to 33% of elementary students indicated that they never use journals. Similarly, 77% 
of middle school students and 54% of elementary students said they never use portfolios. 

Overall, the elementary students reported more frequent use of group assessments than middle 
school students. The difference was largest when comparing the 5*h grade to the 7^* grade. About 61% 
of 5th grade students said that group assessments occurred at least once a month, while only 39% of 7th 
graders reported the use of group assessments this frequently. A similar result occurred with respect to 
the use of manipulatives on assessments, 65% of 5th graders and 41% of 7th graders said they use them 
once a month or more. Overall, these percentages are similar to those the students reported for 
instructional use. 

Impact of Various Features of MSPAP . Teachers and principals were also asked to indicate the 
extent to which various aspects of MSPAP positively influenced their (die teachers’) classroom 
activities. As shown in Table V.24, the aspects of MSPAP that were addressed include the format of 
MSPAP tasks, the content and skills assessed by MSPAP, MSPAP’s integration across subject areas, 
MSPAP’s scoring procedures, and MSPAP’s results. 
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Table V.24 Percentage of Teachers and Principals Indicating the Extent of Impact on 
Classroom Activities for Various Aspects of MSPAP 



MSPAP Aspect 


Moderate or Great Amount of 
Positive Impact 


Teachers 


Principals 


Format of MSPAP Tasks 


61% 


88% 


Content and Skills Assessed by MSPAP 


67% 


93% 


MSPAP’s Integration Across Subjects 


64% 


80% 


MSPAP’ s Scoring Procedures 


51% 


77% 


Results of MSPAP 


49% 


86% 



In general, the majority of both the teachers and principals reported a moderate or great amount of 
positive impact for each of the aspects of MSPAP. However, the percentages were larger for principals 
than teachers. These results are similar to the previously reported results for the overall question 
pertaining to the impact of MSPAP. 

Elementary and middle school principals tended to respond similarly, whereas teachers tended to 
differ across the grade levels. Across the various features of MSPAP, a larger percentage of 3^ and 5^ 
grade teachers, as opposed to teachers in other grade levels, chose the categories “moderate” and “great” , 
whereas die 7*h grade teachers were least likely to choose these categories. As an example, for the 
questions regarding the content and skills assessed by MSPAP and MSPAP’ s scoring procedures, 75% 
and 60% of the 3 rd and 5 th grade teachers chose the moderate and great categories, respectively, 
whereas, 51% and 28% of the 7*h grade teachers chose them, respectively. 

Preparing Students for MSPAP . Principals, teachers, and 5*^ and 8^ grade students were asked to 
respond to questions in two major categories of test preparation methods: a) direct test preparation 
methods including improving student test taking skills, improving student motivation, providing practice 
with similar tasks just prior to die assessment, and b) instructional methods such as changing and/or 
improving instruction throughout the year. Off-grade teachers as well as on-grade teachers were asked 
questions about preparing students for MSPAP to determine die extent to which MSPAP has had an 
effect on both on- and off-grade teachers’ areas of practices. 

As summarized in the Table V.25 below, the majority of the teachers, students, and principals 
indicated, to a moderate or great extent, that there is a focus on improving student test taking skills, 
improving student motivation, and using MSPAP public release tasks or other performance-based tasks 
as test preparation materials. Overall, students and teachers responded similarly. However, principals 
reported that their teachers used these types of test preparation activities more often than what the 
teachers reported. 
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With respect to making changes in instruction to help students prepare for MSPAP, the majority of 
die teachers, students, and principals indicated, to a moderate or great extent, that there has been an 
increased use of MSPAP-like tasks in instruction. The majority of die teachers and principals also 
indicated that the match between the content of instruction and the content of MSPAP has increased. 
More importandy, most of the teachers and principals (89% and 93%, respectively) reported that there 
has been a general improvement in instruction throughout the year. Across the test preparation activities, 
teachers and principals were more in agreement with the statement that teachers improved instruction 
throughout the year than with any of the other test preparation activities. 



Table V.25 Percentage of Teachers. Students, and Principals Indicating a Moderate or Great 
• Use of Various MSPAP Preparation Activities 



Direct Test Preparation Methods 


Teacher 


Student 


Principal 


Improve student test-taking skills 


77% 


70% 


93% 


Improve student motivation 


69% 


82% 


| 93% 


Use of MSPAP-like tasks (e.g., release tasks) 


63% 


68% 


96% 


Instructional Methods 


Increase use of MSPAP-like tasks 


76% 


71% 


94% 


Increase match between content of MSPAP and 
content of instruction 


73% 


— 


83% 


General improvement of instruction 


89% 


— 


93% 



The results for elementary and middle school principals were similar. However, there were several 
differences in die results for teachers across grade levels. The teacher results for the test preparation 
activities related to improving student test-taking skills and improving student motivation were 
somewhat similar across grades, except between die elementary on-grade (3 rd and 5 th ) teachers and the 
middle school off-grade (7*) teachers. About 83% of 3 rd and 5 ^ grade teachers focused on improving 
test-taking skills to a moderate or great extent compared to 67% of the 7^ grade teachers. S imilar ly, 
about 84% of 3 rd and 5*h grade teachers focused on improving student motivation to a moderate or great 
extent as compared to only 42% of the 7^ grade teachers. In these two instances, the percentages for the 
other grade level teachers were between these two extremes. 

In terms of using MSPAP public release tasks or similar tasks as test preparation materials, a large 
percentage of 3 rd and 5 th grade teachers (81%) reported a moderate or great amount of use. The 
percentages for all the other grade-levels were lower (49% for 2 nd and 4^* grade, 36% for 7^ grade, and 
63% for 8th grade). 
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A higher percentage of elementary school teachers (both on- and off-grade) than middle school 
teachers reported that, to a moderate or great extent, they were improving instruction throughout the year 
and increasing the match between the content of instruction and content of MSPAP. Approximately 94% 
of the elementary teachers and 79% of the middle school teachers reported they have improved 
instruction throughout the year, and 78% of the elementary teachers and 58% of the middle school 
teachers indicated that they have increased the content match between instruction and MSPAP. As 
previously indicated, the test preparation activity that was most common for teachers was the general 
improvement of instruction throughout the year. 

Also, there were several differences between the 5^* and 8*b grade student responses to these 
questions. For instance, 75% of 5*b grade students versus 65% of 8*b grade students said that their math 
teacher gave them tips on how to take MSPAP (i.e., improved their test-taking skills). In addition, 91% 
of 5*h graders indicated that their math teacher talked to them about doing their best on MSPAP (i.e., 
motivating students) compared to only 72% of 8^ graders. Student responses to the questions regarding 
use of MSPAP-like math activities prior to the assessment and throughout the year were similar across 
grade levels. 

Impact of MSPAP on Professional Development Activities 

This section describes the content and extent of teacher professional development activities and the 
extent of principal professional development activities that are directly related to MSPAP and the 
MLO’s. 

Content of Teacher Professional Development Activities . Teachers were asked questions 
regarding die extent to which professional development activities for the 1996-97 school year, including 
the g limm er of 1996, addressed the MLO’s, Maryland Curriculum Framework, and specific topics related 
to MSPAP. As shown in the Table V.26, approximately half of the teachers indicated that the MLO’s 
and die Maryland Curriculum Framework were addressed to a moderate or great extent. Further, the 
majority of the teachers indicated that the purpose of MSPAP, format of MSPAP, content and skills 
assessed by MSPAP, preparing students for MSPAP, and using MSPAP results to improve instruction 
was emphasized a moderate or great extent in professional development meetings. The topic related to 
MSPAP that was emphasized the least was explaining results to parents and students. 
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Table V.26 Percentage of Teachers Indicating the Topics Covered in 
Professional Development Activities 



Topics 


Moderate/ 

Great 


Not At 
All 


Maryland Learning Outcomes 


56% 


20% 


Maryland Curriculum Framework 


45% 


29% 


Purpose of MSPAP 


69% 


7% 


MSPAP Task Format 


79% 


6% 


MSPAP Content and Skills 


82% 


4% 


Preparing Students for MSPAP 


78% 


5% 


Use of MSPAP Results to Improve Instruction 


64% 


11% 


Explaining MSPAP Results to Parents and Students 


31% 


36% 



Teachers responded differently across grade levels for the professional development questions 
addressing the MLO’s, purpose of MSPAP, content and skills assessed by MSPAP, preparing students 
for MSPAP, interpreting and using MSPAP results to improve instruction, and explaining MSPAP 
results to students and parents. In general, elementary school teachers indicated that these topics were 
emphasized by professional development activities more often than did middle school teachers, with 7^ 
grade teachers indicating the least frequent emphasis of these topics in staff development activities. 



Extent of Teacher Professional Development Support 

Teacher Professional Development Support . Teachers and principals were asked to indicate the 
extent to which they (their teachers) had the necessary support and/or resources to enable teachers to 
make instruction and assessment changes to belter reflect what is expected of students in MSPAP and the 
MLO’s. As summarized in Table V.27, the majority of the teachers indicated, either to a moderate or a 
great extent, that they had the necessary support and/or resources to enable them to make instructional 
and assessment changes to better reflect what is expected of students in MSPAP and the MLO’s. Only a 
small percentage of teachers indicated that they did not have the necessary support and/or resources to 
make changes in instruction and assessment. Teachers of 3 r ^ and 5^* grade, as compared to the other 
grade level teachers, indicated that they were the most satisfied with the level of support Overall, 
principals, as compared to teachers, indicated that teachers had a greater level of professional 
development support. 
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Table V.27 Percentages of Teachers and Principals Indicating the Extent of 
Teacher Professional Development Support 



Instruction 


Moderate/ Great Amount of Support 


No Support 


Teachers 


Principals 


Teachers 


Principals 


All grades 


63% 


— 


6% 


— 


3 rd and 5th 1 


72% 


92% 


2% 


0% 


2 nd and 4 m 


60% 


90% 


10% 


0% 


gtb 


58% 


86% ] 


9% 


0% 


jm 


58% 


79% 


3% 


0% 


Assessment 










All grades 


56% 1 


— 


10% 


— 


3 rd and 5“ 


67% 


86% 


6% 


0% 


2nd and 4“ 


52% 


81% 


12% 


2% 


! P 


49% 


86% 


13% 


0% 


— Tffi 


47% 


75% 


8% 


4% 



Frequency of MSPAP Teacher Professional Development Activities . Teachers were also asked to 
indicate the number of hours that they would spend in professional development meetings that are 
directly related to MSPAP (e.g., meetings focusing on die nature and purpose of MSPAP, MSPAP tasks, 
MSPAP scoring, and MSPAP results). As summarized in the Table V.28 below, 32% of the teachers 
indicated that they would spend over 16 hours in professional development meetings that are directly 
related to MSPAP within the 1996-97 instructional year, including the summer of 1996. Approximately 
equal percentages indicated that they would spend between 6-15 hours and between 1-5 hours. Only 7% 
indicated that they would spend no time in such professional development meetings. Elementary school 
teachers, in particular, on-grade teachers (3 rd and 5 th ) indicated the most amount of time and 7 th grade 
teachers indicated the least amount of time in MSPAP-related professional development activities. 



Table V.28 Number of Hours of MSPAP Professional Development Activities 



Grade 


Hours 


0 


1-5 


6-15 


16-35 


>36 


All 


7% 


29% 


32% 


20% 


12% 


3 rd and 5 th 


4% 


21% 


31% 


27% 


18% 


2 nd and4tn 


6% 


30% 


35% 


19% 


10% 


“gffi 


12% 


32% 


31% 


19% 


7% 





12% 


45% 


30% 


7% 


7% 



Extent of Various Types of Teacher Professional Development Support . Teachers and principals 
were asked to indicate the extent to which teachers had die necessary support and/or resources in a 
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□umber of areas to enable them (their teachers) to make changes in their classroom activities to better 
reflect what is expected of students in MSPAP and the MLO’s. As Table V.29 indicates, teachers 
reported that they receive die most amount of support in terms of inservices or workshops, whereas they 
get the least amount of support with direct assistance in the classroom (e.g., demonstration lessons). 
Overall, principals, as compared to the teachers, reported that the teachers receive more support in each 
of the areas. In particular, principals perceived that teachers had more time to collaborate with other 
teachers. 

Table V.29 Percentage of Teachers and Principals Indicating the Extent of Various Types of 

Teacher Professional Development Support 



Type of Support 


Moderate/ Great 
Amount of Support 


No Support 


Teachers 


Principals 


Teachers 


Principals 


Inservices/ Workshops 


65% 


87% 


4% 


1% 


Test Preparation 
Materials 


50% 


85% 


15% 


1% 


Instructional Materials 


48% 


77% 


15% 


2% 


Collaboration with other 
Teachers 


34% 


72% 


17% 


1% 


Planning Time 


24% 


44% 


36% 


15% 


Classroom Assistance 


I 16% 


54% 


59% 


15% 



Teachers across grade levels responded similarly except for the question regarding test preparation 
materials. As might be expected, on-grade teachers (3^, 5^, and 8*) indicated more support with 
regard to having test preparation materials than off-grade teachers. In particular, 62% of the on-grade 
elementary teachers and 49% of die on-grade middle school teachers indicated that they had a moderate 
or great deal of support in terms of having test preparation materials, whereas 40% of the off-grade 
elementary teachers and 38% of the off-grade middle school teachers indicated this same level of 
support Elementary and middle school principals responded similarly to these questions. 

Extent of Principal Professional Development Support . Principals were asked to indicate the 
extent to which they had enough support and information to help them move their schools towards the 
goals of MSPAP. Overall, 85% of the principals indicated, to either a moderate or great extent, that they 
had the necessary support and information about MSPAP to help them move toward the goals of 
MSPAP. Elementary and middle school principals responded similarly to this question. 
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Vl. Confirmatory Factor Analyses and Statistical Tests for the 

Questionnaire Data 

Confirmatory Factory Analyses for the Mathematics Teacher Questionnaire 
Confirmatory factor analyses (CFA; Joreskog, 1969; Joreskog & Sorbom, 1979) allowed for the 
examination of the underlying structure of the teacher questionnaire. The teacher questionnaire was 
designed to provide information about six dimensions. They included teachers’ familiarity with MSPAP, 
teachers’ support for MSPAP, teachers’ instruction and assessment practices, change in teacher’s 
instruction and assessment practices, MSPAP’s impact on instruction, and professional development 
support for teachers. Subsets of items were grouped according to the following 1 1 areas (i.e., measures) 
to reflect the six dimensions: 

(1) MSPAP Familiarity - General (teachers’ general familiarity with MSPAP), 

(2) MSPAP Familiarity - Results (teachers’ familiarity with MSPAP results), 

(3) Support MSPAP - General (teachers’ general support for MSPAP), 

(4) Support MSPAP - Instruction (teachers’ support for MSPAP for instructional purposes), 

(5) Current Math Instniction/Assessment - LO (emphasis on learning outcomes in instruction 
and assessment), 

(6) Current Math Instruction/Assessment - PT (emphasis on reform-oriented problem types in 
instruction and assessment), 

(7) Change Math Instruction/Assessment - LO (change in emphasis on learning outcomes in 
instruction and assessment), 

(8) Change Math Instruction/Assessment - PT (change in emphasis on reform-oriented problem 
types in instruction and assessment), 

(9) MSPAP’s Impact (MSPAP’s impact on instruction and assessment), 

(10) Professional Development Support - MSPAP (professional development activities related to 
MSPAP), and 

(11) Professional Development Support - Amount (amount of professional development 
activities). 

Teacher mean scores were obtained for each of these eleven subsets of items. The majority of the items 
on the questionnaire had a four-point Likert scale. For those items that had more than a four point scale, 
the responses were recoded to a four-point scale. Teacher data were excluded for those cases in which 
teachers had left blank more than 25% of the items on any one of the eleven subsets of items. Based on 



the intercorrelations among the items and the item-to- total score correlations, a small number of items 
were deleted from their respective subsets. For example, a few items were deleted due to low item-to- 
total score correlations. Figure VI. 1 provides the final set of items for each of the subsets and the 
hypothesized dimension underlying each of the subset of items. Coefficient alpha reliability estimates 
for these 1 1 subsets (i.e., measures) for both on- and off-grade data sets ran g ed from .74 to .93. 



Figure VI. 1 Hypothesized Dimensions. Measures, and Mathematics 
Teacher Questionnaire Items 



Dimension/ Measure 


Teacher Questionnaire Item 


MSPAP Familiarity 




General 


To what extent are you familiar with each of the following? 




purpose of MSPAP 




format of MSPAP tasks 




content and skills assessed by MSPAP 




how to prepare students for MSPAP 


Results 


how to interpret and use MSPAP results to improve classroom 
instruction and assessment 




how to explain MSPAP results to students and/or parents 


Support MSPAP 




General 


To what extent do you support or oppose MSPAP? 




To what extent has your support or opposition of MSPAP changed 
over the last few years? 




To what extent do you support or oppose the reporting of MSPAP 
results? 




To what extent do you support or oppose holding schools 
accountable for meeting the performance standards on MSPAP? 


Instruction 


MSPAP is a useful tool for helping me make positive changes in 
my instruction. 




MSPAP is a useful tool for making positive changes in instruction 
for those teachers who are resistant to change. 




Results of MSPAP provide useful information for making 
inferences about school improvement. 


Current Math 
Instruction/Assessment 




Learning Outcomes 


About how often have you asked your students to work on 
classroom activities that emphasize mathematical problem solving 
and reasoning this year (1996-97)? 




About how often have you asked your students to work on 
classroom activities requiring mathematical explanations and 
justifications this year (1996-97)? 




About how much emphasis have you placed on each of the 
following learning outcomes (LO) in vour mathematics instruction 
this year (1996-97)? 




problem solving 




communication ' 




reasoning 




connections 



Figure VI. 1 Hypothesized Dimensions. Measures, and Mathematics 
Teacher Questionnaire Items - Continued 





About how much emphasis have you placed on each of the 
following LO in vour mathematics assessment this vear (1996- 
97)? 




problem solving 




communication 




reasoning 




connections 


Problem Type 


About how often have you used each of the following types of 
problems in your mathematics classroom this year (1996-97)? 




open-ended problems 




problems that take a few days or more to complete 




problems emphasizing relationships among mathematics 
concepts 




problems that integrate other subject areas in math 




problems that apply math to real-life situations 




About how often have you used each of the following types of 
assessment questions and activities this year (1996-97)? 




open-ended problems 




projects/extended activities 




group activities 




use of manipulatives 




written explanations 


Change Math 
Instruction/Assessment 




Learning Outcomes 


How has the emphasis on mathematical problem solving and 
reasoning changed from 1992-93 to 1996-97? 




How has the emphasis on mathematical explanations and 
justifications changed from 1992-93 to 1996-97? 




How has the emphasis for each learning outcome in your 
mathematics instruction changed from 1992-93 to 1996-97? 




problem solving 




communication 




reasoning 




connections 




How has the emphasis for each learning outcome in your 
mathematics assessment changed from 1992-93 to 1996-97? 




problem solving 




communication 




reasoning 




connections 
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Figure VI. 1 Hypothesized Dimensions. Measures, and Mathematics 
Teacher Questionnaire Items — Continued 



Problem Type 


How has tile emphasis for each problem type used in your 
mathematics classroom changed from 1992-93 to 1996-97? 




open-ended problems 




problems that take a few days or more to complete 




problems emphasizing relationships among mathematics concepts 




problems that integrate other subject areas in math 




problems that apply math to real-life situations 




How has tile emphasis for each type of assessment question and 
activity changed from 1992-93 to 1996-97? 




open-ended problems 




projects/extended activities 




group activities 




use of manipulatives 




written explanations 


MSPAP Impact 






About how much has MSPAP influenced your mathematics 
classroom activities this year (1996-97)? 




To what extent has MSPAP influenced you to make positive 
changes in your mathematics instruction? 




To what extent has MSPAP influenced you to make positive 
changes in your mathematics assessment? 




About how often do you ask your students to solve mathematics 
tasks similar to those on MSPAP? 




To what extent have you focused on the following strategies in 
preparing your students for MSPAP? 




increasing the use of MSPAP- like tasks in regular instruction 




increasing the match between the content of instruction and the 
content of MSPAP 


Professional Development 
Support 




Focus on MSPAP 


To what extent did staff development activities address the 
following? 




purpose of MSPAP 




format of MSPAP tasks 




content and skills assessed by MSPAP 




how to prepare students for MSPAP 




how to interpret and use MSPAP results to improve instruction 




how to explain MSPAP results to students/parents 


Amount of Support 


To what extent have you had the necessary support and/or resources 
to enable you to make instructional and assessment changes to better 
reflect what is expected of students in MSPAP and the Maryland 
Learning Outcomes? 




support for instruction 




support for assessment 



76 



O 




77 



A maximum likelihood estimation procedure was used to estimate three hierarchical models using 
AMOS (Arbuckle, 1997). Two sets of analyses were conducted. The first set excluded two teacher 
mean scores. Change Math Instruction-Learning Outcomes and Change Math Instruction-Problem Type, 
whereas, the second set of analyses included these two scores. Teachers answered the questions with 
respect to instructional change only if they were a teacher in Maryland since the 1992-93 school year. 
Thus, the second set of analyses is based on a smaller sample size than the first set of analyses. 

For the analyses excluding the instructional change measures, the first model that was estimated 
provided a test for the hypothesis that one factor accounted for the interrelations among the teacher mean 
scores for nine measures. The second model that was estimated provided a test for the hypothesis that 
four factors accounted for the interrelationships as specified in Figure VI.2. The third model that was 
estimated, the hypothesized model, provided a test for the hypothesis that five factors accounted for the 
interrelationships as specified in Figure VI.3. For the analyses including the instructional change 
measures, similar models were estimated as shown in Figures VI.4 and VI. 5; however, the third model 
included six factors so that one factor would reflect the two instructional change measures. 

The analyses were done for the on-grade levels (3, 5, 8) combined and the off-grade levels (2, 4, 7) 
combined. The sample sizes for the analyses excluding the instructional change measures were 254 for 
the on-grade and 172 for the off-grade. The sample sizes for the analyses including the instructional 
change measures were 178 for the on-grade and 1 12 for the off-grade. 
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Figure VL2. Four-Factor Model Excluding Instructional Change Measure 
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figure VL3. Five-Factor Model Excluding Instructio nal Chanye Measure 
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Figure VIA Four-Factor Model Incl uding Instru ctional Change Measure 
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Figure VL5, Six-Factor Model Including Instructional Change Measure 
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Results for the Analyses Excluding the Instructional Change Measures 

As indicated in Table VI. 1, for die on-grade analyses excluding the instructional change measures, 
the one- factor model and the four-factor model did not fit the data as evidenced by the significant chi- 
square. The five-factor model fit the data as evidenced by the nonsignificant chi-square statistic. Only 
one covariance among the factors was not significant and it was for the relationship between Support 
MSPAP and Current Math Instruction. 



Table VI. 1 Confirmatory Factor Analysis Excluding Instructional/Assessment 
Change Measures - Teacher Questionnaire 





xT~ 


df 


P 


RMSEA 


NFI 


On-grade (n=254) 












1-factor model 


258.407 


27 


.000 


.184 


.686 


4-factor model 


63.053 


21 


.000 


.089 


.923 


5-factor model 


18.859 


18 


.401 


.014 


.977 


Off-grade (n=172) 












1-factor model 


150.488 


27 


.000 


.164 


.747 


4-factor model 


40.702 


21 


.006 


.074 


.931 


5-factor model 


18.362 


18 


.432 


.011 


.969 


On and off grade 












5-factor model 












Constrained 


73.634 


63 


.169 


.020 


.948 


Unconstrained 


37.227 


36 


.412 


.009 


.974 



These analyses were also conducted for the off-grade levels (2, 4, 7), combined. The hypothesized 
models, excl uding the instructional change measures, were estimated to determine whether the 
underlying structure of the teacher questionnaire was similar for the on- and off- grades. The five-factor 
model for the off-grade levels, which excluded the instructional change measures fit the data as 
evidenced by the nonsignificant chi-square statistic in Table VI. 1. All of the covariances among the 
factors were significant 

A third set of analyses was conducted to determine whether the parameters could be constrained 
across the on- and off-grades for the five factor model. The results are provided in Table VI. 1. The 
difference chi-square of 36.407 with 27 df was not significant (p = .107), indicating that file additional 
parameters estimated under the unconstrained model did not improve on model data fit as offered by the 
constrained model. Thus, the parameters could be constrained across the two groups. Table VI.2 
provides the unstandardized regression coefficients, their standard errors, and the significance tests for 
the five-factor model with the parameters constrained across the on- and off-grades. The l’s in the 
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column for the unstandardized regression coefficients denote the necessary constraints to attain model 
identification. 



Table VI.2 Regression Coefficients and Significance Tests for Confirmatory Factor 
Model with Five Factors - On and Off made levels (Parameters constrained"! - 

Teacher Questionnaire 



Dimension and Measure 


Unstandardized 
Regression coefficients 


SE 


t 


MSPAP Familiarity 








General 


1.000 






Results 


1.546 


.115 


13.484* 


Support MSPAP 








General 


1.000 






Instruction 


1.304 


.174 


7.488* 


Current Math Instruction/ 
Assessment 








Learning outcomes 


.835 


.056 


14.965* 


Problem types 


1.000 






MSPAP Impact 


1.000 






Professional Dev. Support 








MSPAP 


1.000 






Amount 


.777 


.094 


8.305* 



Note: *p< .01 



Results for the Analyses Including the Instructional Change Measures 

Similar results were found for the on-grade analyses that included the instructional change measures. 
The one-factor model and the four-factor model did not fit the data as evidenced by the significant chi- 
square statistics in Table VI.3. The six-factor model fit the data as evidenced by the nonsignificant chi- 
square statistic. The only covariance among the factors that was not significant is for the relationship 
between Support MSPAP and Current Math Instruction. 
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Table VI.3 Confirmatory Factor Analysis Including Instructional/ Assessment 
Change Measures - Teacher Questionnaire 







df 


C 


RMSEA 


NFI 


On-grade (n=l 78) 












1 -factor model 


319.030 


44 


.000 


.188 


.605 


4-factor model 


183.384 


38 


.000 


.147 


.773 


6- factor model 


34.777 


30 


.251 


.030 


.957 


Off-grade (n=l 12) 












1 -factor model 


232.182 


44 


.000 


.196 


.629 


4-factor model 


140.088 


38 


.000 


.156 


.776 


6-factor model 


41.682 


30 


.076 


.059 


.933 


On and off grade 












6-factor model 












Constrained 


135.607 


96 


.005 


.038 


.905 


Unconstrained 


76.500 


60 


.074 


.031 


| .947 



These analyses were also conducted for the off-grade levels (2, 4, 7) combined. The hypothesized 
models, including the instructional change measures, were estimated to determine whether the 
underlying structure of the teacher questionnaire was similar for the on- and off-grades. Similar to the 
on-grade levels, the one- and four-factor models for the off-grades did not fit the data as evidenced by 
the significant chi-square statistic in Table VI.3. The six-factor model for the off-grade levels did fit the 
data as evidenced by die nonsignificant chi-square statistic in Table VO. All of the covariances among 
the factors were significant 

Another set of analyses were conducted to determine whether the parameters could be constrained 
across the on- and off-grades for the six factor model, including instructional change. The results are 
provided in Table VI.3. The difference chi-square of 59.107 with 36 df was significant (p=.009), 
indicating that the additional parameters estimated under the unconstrained model improved on model 
data fit. Thus, the parameters cannot be constrained across the two groups. Table VI.4 provides the 
unstandardized regression coefficients, their standard errors, and the significance tests for the six-factor 
model for the on-grade levels and the off-grade levels. 
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Table VI .4 Regression Coefficients and Significance Tests for Confirmatory Factor 
Model with Six Factors - On-grade and Off- grade Levels - Teacher Questionnaire 



Dimension and Measure 


Unstandardized 
Regression coefficients 


SE 








On- 

grade 


Off- 

grade 


On- 

grade 


Off- 

grade 


On- 

grade 


Off- 

grade 


MSPAP Familiarity 














General 


1.000 


1.000 










Results 


1.509 


1.946 


.204 


.269 


7.406* 


7.228* 


Support MSPAP 














General 


1.000 


1.000 










Instruction 


1.200 


1.303 


.201 


.277 


5.958* 


4.703* 


Current Math Instruction/ 
Assessment 














Learning outcomes 


.830 


.628 


.080 


.080 


10.412* 


7.858* 


Problem types 


1.000 


1.000 










Change Math Instruction/ 
Assessment 














Learning outcomes 


1.088 


.770 


.097 


.094 


11.240* 


8225* 


Problem types 


1.000 


1.000 










MSPAP Impact 


1.000 


1.000 










Professional Dev. Support 














MSPAP 


1.000 


1.000 










Amount 


.679 


.929 


.144 


.151 


4.727* 


6.175* 



Note: *p<.01 



In general, these results su gg est that the underlying structure of die teacher questionnaire items for 
the off-grade levels is similar to the structure for the on-grade levels when excluding the instructional 
change measures. When including the instructional change measures, the factor structure for the on- and 
off-grade levels is similar, however the relationship between the measures and the factors differ across 
the on- and off-grades to some extent 
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Multivariate Analysis of Variance for the Questionnaire Data 



Results for the Teacher Questionnaire 

The teacher questionnaire data were analyzed with a one-way multivariate analysis of variance, with 
the between- subj ects effect being the grade and the dependent measures being the teacher composite 
mean scores on the dimensions: MSPAP Familiarity, Support MSPAP, Current Math Instruction, Change 
Math Instruction, MSPAP Impact on Instruction, and Professional Development Support Descriptive 
data for die dependent measures are provided in Table VI.5. The range on die questionnaire item scale is 
1-4, with the more positive responses being at the upper end of the scale. Overall, the mean scores were 
at the upper end of the score scale. 



Table V1.5 Descriptive Data for the Six Dimensions- 
Teacher Questionnaire 



Dimension 




Off-Elem 

( 2 nd/ 4 th) 

(n=81) 


On-Elem 

( 3 rd /5 th) 

(n=120) 


Off-Middle 

(7 th) 

(n=3 1) 


On-Middle 

(gth) 

(n=58) 


MSPAP 


mean 


3.230 


3393 


2.930 


3.175 


Familiarity 


sd 


.572 


.566 


.673 


.562 


Support 


mean 


2.639 


2.549 


2.544 


2.508 


MSPAP 


sd 


.603 


.604 


.550 


.610 


Current Math 


mean 


3.140 


3.296 


2.916 


2.993 


Instruction/ 

Assessment 


sd 


.493 


393 


360 


.486 


Change Math 


mean 


3.029 


3.181 


2.962 


2.945 


Instruction/ 

Assessment 


sd 


.401 


.459 


.309 


.351 


MSPAP 


mean 


2.964 


3.255 


2.628 


2.818 


Impact 


sd 


.605 


.586 


.509 


.692 


Professional 


mean 


2.866 


3.080 


2.427 


2.704 


Dev Support 


sd 


.621 


.575 


.616 


.756 



The multivariate test was significant at p <.001 (Wilkes’ Lambda, F (18, 795) =3.568, p < .001.). 
j Table V1.6 provides a summary of the results of the univariate analyses. As indicated in the table, there 

were significant grade differences for five of the dimensions: MSPAP Familiarity, Current Math 
| Instruction, Change Math Instruction, MSPAP Impact on Instruction, and Professional Development 

Support. 
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Table VI.6 Univariate ANOVA’s for the Six Dimensions- 
Teacher Questionnaire 



Dimension 


df 


F 


b 




MSPAP Familiarity 


3 


5.956 


.001 


.049 


Support MSPAP 


3 


.623 


.601 


.004 


Current Math Instruction/ 
Assessment 


3 


9.850 


.000 


.084 


Change Math Instruction/ 
Assessment 


3 


5.730 


.001 


.047 


MSPAP Impact 


3 


12.702 


.000 


.108 


Professional Dev. 
Support 


3 


10.818 


.000 


.092 



Tukey HSD post-hoc analyses were conducted to determine, for each of the five dependent measures, 
which differences between composite mean scores were significant Table VI. 7 provides the results of 
the post-hoc analyses. In general, an e xaminati on of the table indicates that composite mean scores for 
elementary on-grade teachers were significantly greater than composite mean scores for middle on- and 
off-grade teachers. For example, elementary on-grade teachers, as compared to middle on- and off-grade 
teachers, were more likely to indicate that they place a greater emphasis in their mathematics classrooms 
on the learning outcomes and reform oriented problem types as evidenced by die composite mean 
differences for the dimension. Current Math Instruction. Elementary on-grade teachers, as compared to 
middle on-grade teachers, were also more likely to indicate that their emphasis on the learning outcomes 
and reform oriented problem types is greater than what it was a few years ago as evidenced by the mean 
differences for the variable. Change Math Instruction. Further, elementary on-grade teachers, as 
compared to middle on- and off-grade teachers, were more likely to indicate that MSPAP had a greater 
impact on their mathematics instruction and that they had received more professional development 
support regarding MSPAP as evidenced by the mean differences for the dimensions, MSPAP Impact and 
Professional Development Support, respectively. As indicated in Table VI.6, however, the adjusted 
value is relatively small for each of the significant variables indicating that grade accounts for only a 
small percentage of the variance. 
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Table V1.7 Tukev HSD Post-Hoc Analyses - Teacher Questionnaire 



Dimension 


Contrast 


Mean 

Difference 


SE 


B 


MSPAP Familiarity 


3/5 vs 7 


.463 


.117 


.000 


Current Math Instruction/ 


3/5 vs 7 


.380 


.089 


.000 


Assessment 


3/5 vs 8 


.303 


.070 


.000 


Change Math Instruction/ 


3/5 vs 2/4 


.151 


.059 


.049 


Assessment 


3/5 vs 7 


.218 


.082 


.040 




3/5 vs 8 


.236 


.065 


.002 




2/4 vs 7 


.336 


.128 


.043 


MSPAP Impact 


3/5 vs 2/4 


.291 


.087 


.005 




3/5 vs 7 


.627 


.122 


.000 




3/5 vs 8 


.437 


.097 


.000 




2/4 vs 7 


.438 


.133 


.006 


Professional Development 


3/5 vs 7 


.652 


.127 


.000 


Support 


3/5 vs 8 


3 76 


.101 


.001 



There were few differences between mean scores for elementary on- and off-grades and when these 
differences occurred they were small. For example, elementary on-grade teachers, as compared to 
elementary off-grade teachers, were more likely to indicate that their emphasis on the learning outcomes 
and reform oriented problem types is greater than what it was a few years ago, as evidenced by the mean 
differences for the variable, Change Math Instruction. However, the mean difference was small (e.g., 
.151 , p = .049). 

In summary, elementary on-grade teachers as compared to middle on- and off-grade teachers 
indicated that their instruction was more aligned to the content and format of MSPAP and that they have 
had more professional development support related to MSPAP. Further, there were only a few 
differences between elementary on- and off-grade teacher results and no difference between middle on- 
and off-grade teacher results. In general, this suggests that, within school type (elementary or middle), 
teachers of grades that were not tested reported similar results as teachers of grades that were 
administered MSPAP. 



Results for the Principal and Student Questionnaire 

Elementary and middle school principals were asked to respond to some of the same items as in the 
teacher questionnaire. Table VI. 8 provides elementary and middle school principal mean scores on four 
of the dimensions discussed above: MSPAP Familiarity, Support MSPAP, MSPAP Impact, and 
Professional Development Support This table also provides corresponding mean scores for the teachers. 
It should be noted that the mean scores for the teachers in this table are somewhat different than the 
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mean scores provided in Table VI.5. This is because the scores in Table VI. 8 are based on only the items 
that were die same for die principals and the teachers. For the dimensions, MSPAP Familiarity and 
Support MSPAP, the items were the same for both teachers and principals. For the dimensions, MSPAP 
Impact and Professional Development Support, the principals had fewer items than the teachers and 
consequendy the teacher means in Table VI. 8 are based on a smaller number of items than those reported 
in Table VI.5. 
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Table VI.8 Descriptive Data for the Dimensions - Teacher. Principal, and Student Questionnaire 



Class (Students) 


4 oe || 

o 






2.603 

.302 






U 

<S 

T3 

h 2 

t§ 

c o 

i § 

-H i 

|i 

<2 | 

m 8 
m -o 
•o 3 

§s 

a CQ 


Off- Middle 
(7 th ) 
(n=35) 






2.490 

.227 






On-Elem 

(5*) 

(n=60) 






2.759 

.271 






s s it 
§ * 






2.672 

.314 






Note; The teacher sample sizes for the dimension, Current Math Instruction and Assessment, are 41, 60, 3 
elementary, off-middle, and on-middle, respectively. This is because only teachers with corresponding ck 
this analysis. 


Principal 


a> /-n 

1 5 

s & 


3.475 

.369 


3.147 

.423 




3.191 

.438 


3.080 

.657 


p in 

I IT 

w e 


3.613 

.400 


3.109 

.557 




3.464 

.535 


3.196 

.546 


Teacher 


o 

ti 

a a VD 
S S2, L 

a* 


3.175 

.562 


2.508 

.610 


2.714 

.554 


2.879 

.701 


2.560 

.937 


« 

ic? 

O 


2.930 

.673 


2.544 

.550 


2.652 

.481 


2.734 

.577 


2.581 

.776 


i 4^ o' 

A ■B II 

S C-vS 


3.393 

.566 


2.549 

.604 


3.147 

.472 


3.308 

.645 


3.008 

.791 


S 5 oo 

2^ H. 

IS CN C 

o 


3.230 

.572 


2.639 

.603 


2.8569 

.486 


3.031 

.655 


2.784 

.869 




mean 

sd 


mean 

sd 


mean 
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mean 

sd 


mean 

sd 




Dimension 


MSPAP 

Familiarity 


Support 

MSPAP 


Current Math 
Instruction/ 
Assessment 


MSPAP 

Impact 


Professional 
Dev Support 



CM 

CD 



CD 
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A one-way multivariate analysis of variance was conducted, with the between-subjects effect being 
the school type and the dependent measures being the composite mean scores on the four dimensions of 
the principal questionnaire. The multivariate test was not significant (Wilkes* Lambda, F (4, 77) = 2.24S, 
p — .072). This result suggests that elementary and middle school principals are similar with respect to 
their familiarity with MSPAP, their support of MSPAP, the extent to which they think MSPAP has had 
an impact on instruction, and the extent to which they think their teachers have received professional 
development support related to MSPAP. 

In general, the principal composite mean scores were higher than the teacher composite mean scores 
on the dimensions as indicated in Table VI. 8. A one-way multivariate analysis of variance was 
conducted, with the between-subjects effect being teacher/principal and the dependent measures being 
the composite mean scores on the four dimensions of the principal questionnaire. The multivariate test 
was significant (Wilkes’ Lambda, F (4, 367) - 16.510, p < .001). Table VI.9 provides a summary of the 
results of the univariate analyses. All of the univariate tests were significant. Both elementary and 
middle school principals, as compared to elementary and middle school teachers of mathematics, 
indicated that they were more familiar with MSPAP, that they were more supportive of MSPAP, that 
MSPAP had a greater impact on classroom instruction, and that teachers were receiving more 
professional development support related to MSPAP. It should be noted, however, that the adjusted r 2 
values were relatively small. 



Table VI.9 Univariate ANOVA’s for the Four Dimensions- Teacher vs. Principal 



Dimension 


df 


F 


£ 




MSPAP Familiarity 


1 


20.310 


.000 


.049 


Support MSPAP 


1 


58.581 


.000 


.134 


MSPAP Impact 


1 


12.783 


.000 


.031 


Professional Dev. 
Support 


1 


11.848 


.001 


.028 



Students in 4^*, 5*h, 7^, and 8^* grade were also asked to respond to some of the same items as in 
the teacher questionnaire related to the dimension. Current Math Instruction. Class composite mean 
scores for each of the grades were obtained on this dimension and are provided in Table VI.8. A one- 
way univariate analysis of variance, with the between-subjects effect being the grade level was 
conducted on the class data. The univariate test was significant as indicated in Table VI. 10. 
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Table VI. 1 0 Univariate ANOVA’s for One Dimension- 
Teacher and Student Questionnaire 





Teacher 


Class (Students) 


Dimension 


df 


F 


E 




df 


F 


fi 




Current Math Instruction/ 
Assessment 


3 


15.550 


.000 


.123 


3 


7.841 


.000 


.089 



Tukey HSD post-hoc analyses were conducted to determine which differences between mean scores 
were significant Table VI. 1 1 provides the results of die post-hoc analyses. As indicated in the table, 
elementary on-grade students (5^) and off-grade students (4th) were more likely to indicate that a 
greater emphasis was placed on the learning outcomes and reform-oriented problems than off-grade 
students (7th). Further, on-grade elementary school students (5th) were more likely to indicate that a 
greater emphasis was placed on the learning outcomes and reform-oriented problems than middle on- 
grade students (8th). it should be noted, however, that the mean differences, although significant, are 
relatively small given the 4-point scale. Further, die adjusted r^ value of .089 is relatively small, 
indicating that approximately 9 % of file variance in the Current Math Instruction variable is accounted 
for by grade level. 



Table VI. 1 1 Tukev HSD Post-Hoc Analyses - 
Teacher and Student Questionnaire 







Teacher 






Class (Students) 




Dimension 


Contrast 


Mean 

Diff 


SE 


B 


Contrast 


Mean 

Diff 


SE 


B 


Current Math 


3/5 vs 2/4 


.239 


.071 


.004 


4 vs 7 


.177 


.061 


.019 


Instruction/ 


3/5 vs 7 


.500 


.099 


.000 


5 vs 7 


.261 


.057 


.000 


Assessment 


3/5 vs 8 


.422 


.079 


.000 


5 vs 8 


.159 


.050 


.008 



In general, the composite mean scores for classes on this dimension were consistently lower than the 
teacher composite mean scores. A one-way univariate analysis of variance, with the between- subj ects 
effect being the class/teacher was conducted on the data. The univariate test was significant, F (1, 376) = 
25367, p <.000. This suggests that teachers, as compared to students, were more likely to indicate that 
their mathematics classrooms had a greater emphasis on the learning outcomes and reform oriented 
problem types. Similar to the previous results, the adjusted r^ value of .061 is relatively small. 
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indicating that approximately 6% of die variance in the Current Math Instruction variable is accounted 
for by the type of respondent (teacher vs. class of students). 



VII. MSPAP Performance Gains From 1993-97 in Relation to 
“MSPAP Impact” and School Characteristic Variables 



Modeling Differences in School Performance Over Time 

Random coefficient or growth modeling was used to examine mathematics performance on MSPAP 
from 1993 to 1997 in relation to two dimensions from die teacher questionnaire and die variable, percent 
free or reduced lunch. Only two dimensions, MSPAP Impact and Current Math Instruction, were used 
because of the relatively s mall school sample size. In addition, these two dimensions were considered to 
be more relevant than the other dimensions for examining die relationship between change and teachers’ 
perceptions. 

The advantages of using growth curve methodologies to analyze change has been discussed in the 
literature (c.f., Rogosa & Willet, 1985; Willet & Sayer, 1994; Rogosa, 1987). These methodologies are 
particularly well suited for studying processes that consider change as continuous with individual 
differences in the pattern of change (e.g., initial level and rate of growth). Further, these methodologies 
allow for studying individual differences and identifying factors that affect the trajectory of change. This 
type of analysis can not be modeled by time-specific comparisons involving group-level (e.g., means) 
differences. 

Figure VII. 1 illustrates the differences in initial mean MSPAP performance and changes in mean 
MSPAP performance from 1993 to 1997 for die sample of schools in the present study. Since percent 
free or reduced lunch was found to correlate significantiy with 1993 MSPAP math performance, the plots 
are presented for three subgroups of this variable (i.e., lower 3^, middle 3 r< ^, and upper 3^) to reduce 
the number of lines in any one graph. As can be seen, there are differences among the schools in terms 
of their initial MSPAP math performance and their change over time. 
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Mean School Performance Index Mean School Performance Index Mean School Performance Index 



Figure VII. 1 Change in School Performance Index Over Time 
bv Percent Free Lunch Percentiles 
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In order to model individual differences in change and assess the correlates or predictors of change, 
two levels of statistical modeling are required: Level 1 - within individual schools, trends across die 
repeated measurements are modeled; and Level 2 - across schools, die parameters from the model of 
individual differences in change at Level 1 are modeled in relation to other factors. At Level 1, growth 
models analyze the repeated measurements of test scores, analyze die relationship between time (year) 
and test score levels, and estimate a reference status (intercept) and rate of change (slope) for each 
school. The parameters from the model at Level 1 (intercepts and slopes) are then modeled in relation to 
school characteristics to explore the relationship between the reference status of MSPAP performance 
and changes in performance with the two dimensions of the teacher questionnaire and the variable, 
percent free or reduced lunch. 

One recently developed random coefficient model, the latent variable growth model, was used in this 
study. Latent variable growth models estimate random coefficient or growth models within the 
framework of structural equation models (c.f., Meredith & Tisak, 1990; McArdle & Epstein, 1987; 
Muthen, 1994) and offer several advantages over more traditional growth curve methodologies. First, 
these models allow die researcher to reflect an underlying theory and evaluate latent variables or 
theoretical constructs in the processes underlying change. Another advantage to this procedure is that 
more general analyses can be conducted. Multiple outcome variables with different growth processes 
can be modeled, individual variability in rates of growth can be modeled as a function of mediating 
factors, multiple indicator latent factors reflecting theoretical constructs can be defined to model 
measurement error, and differences in growth can be examined as a function of either a treatment 
intervention or some other grouping variable. Latent variable growth models can be estimated in 
currently available structural equation modeling software such as LISREL (Joreskog & Sorbom, 1994) or 
AMOS (Arbuckle, 1997). These programs provide maximum likelihood estimation of parameter 
estimates, asymptotic standard errors, goodness-of-fit statistics, and the ability to compare nested models 
which reflect competing hypotheses. In the present study, AMOS was used. 

Figure VII.2 presents the Level 1 latent variable growth model for the present study. This model 
involves the outcome variable, MSPAP mathematics standard score, measured at five timepoints. In 

order to translate the growth model into the framework of structural equation modeling, the school- 

< 

specific random coefficients (intercepts and slopes from Level 1) are each modeled using two latent 
factors: 1) a factor representing a reference status of MSPAP math performance (intercept), and 2) a 
factor which corresponds to die rate of change in MSPAP math performance over time (slope). The mean 
of these factors represent group level estimates (Level 2) of the intercepts and slopes, respectively, and 
the variance of these factors reflects the school differences or random effects that exist around these 



group level parameters. Larger variances reflect increased variability or less similarity in intercept and 
slopes among die schools. 



Figure VIL2. Level I Latent Variable Growth Model 
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As can be seen from the figure, the Level 1 model has the format of a measurement or confirmatory 
factor analysis model in structural equation modeling with restrictive loadings: Y = Ar| + e, where Y are 
the original measurements over time, q is a vector of latent variables (intercept and slope parameters), 
A is a matrix of regression coefficients relating the slope and intercept factors to the Y measurements, 
and 6 is a vector of residuals representing variance not accounted for due to time specific factors not 
included in the model or random error. In addition, an association between die intercept and slope 
factors is assumed and indicated by the curved bi-directional arrow in the figure. 

The meaning of the intercept factor depends on the scaling of the time variable for the slope factor, 
and the scaling of the slope factor is determined by die factor loadings or regression coefficients relating 
the slope factor to the observed measurements. For example, to reflect a simple linear pattern, the 
regression coefficients could be constrained to be 0, 1, 2, 3, and 4 for the variables, 1993 MSPAP 
performance to 1997 MSPAP performance. Under this scaling, the intercept could be interpreted as 
MSPAP initial status of schools since time 0 corresponds to 1993 performance. However, it is also 
possible to estimate coefficients or constrain the parameters to some other pattern. In this figure, the 
pattern is 4, 3, 2, 1, and 0. Since time 0 is associated with 1997 MSPAP performance, the intercept 
factor is interpreted as 1997 MSPAP status and a decrease in performance would be expected from 1997 
to 1993. This scaling was adopted because other school related information was collected in 1997 and 
introduced into the analysis to explain variations in the 1997 MSPAP performance and rates of change 
among schools. The intercept factor will be referred to as 1997 MSPAP performance hereafter. 

The structure or distribution of the residuals (Level 1 error models) is defined through constraints on 
the parameters of the error variance-covariance matrix. The classical assumption of homoscedastic 
independent errors can be defined by constraining the diagonal elements (variances) of the error variance 
covariance matrix to be equal over time and off-diagonal elements (covariances) fixed at 0. This 
assumption can be relaxed by allowing the variances to vary over time and/or estimating a certain pattern 
to the error variances and covariances (e.g., compound symmetry or adjacent error covariances 
estimated). In addition, all error variances and covariances can be estimated as in a fully parameterized 
or unstructured error matrix. In Figure VTI.2, independent but unequal error variances are assumed. 

In order to estimate group level estimates of the intercept and slope latent variables for the Level 2 
model, means for the latent variable intercepts and slope factors must be estimated. The general 
covariance structure model accommodates such a parameterization and is often used when analyzing 
longitudinal data or multiple populations. In order to estimate these types of models, the general 
covariance structure model includes an intercept term as follows: Y = x + Aq + e, where x is a vector of 
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intercepts and is the E[Y] when T| = 0, and all other model parameters are defined as before. Note that 
t’= 0 when deviations from means are analyzed. 

Table VII. 1 presents the results from estimating the Level 1 model in Figure VH.2 for 86 schools (1 
aberrant pattern of performance over time was detected and deleted for the growth curve analyses). The 
chi-square statistic for model-data-fh was 8.162 with 9 df (p=.518) indicating that foe null hypothesis 
that foe variance-covariance matrix implied by foe model in the table equals foe observed variance- 
covariance matrix could not be rejected. As can be seen, foe 1997 MSPAP performance (intercept 
factor) across foe schools was 521.612 with a significant mean rate of change (slope factor) of -2.701, 
although foe rate of change was modest given the scale of foe test scores. Recall that foe rate of change 
is associated with a decrease in performance from 1997 to 1993. Thus, this result suggests that there was 
a significant increase in performance from 1993 to 1997. The variances for 1997 MSPAP performance 
and rate of change indicate significant variability in these parameters across foe school. In addition, foe 
covariance between 1997 MSPAP performance and rate of change was not significant (r = -.050). In 
order to investigate this last finding further, an analysis in which 1993 MSPAP performance was foe 
reference point was examined. This analysis revealed a significant negative covariance between 1993 
MSPAP performance and rate of change (r * -.404) indicating that higher rates of change were associated 
with lower initial performance in 1993. This suggests that foe rate of change is more similar for schools 
in 1997 than in 1993 and this may be due to foe observed decrease in variability in 1997 school 
performance as compared to 1993. 
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Table VP. 1 Results for the Level 1 Growth Model 



Measure and variable 


Estimates 


SE 


I 


Regression Coefficients: 








Math93<- 1997 Performance 


1 






Math94<- 1997 Performance 


1 






Math95<- 1997 Performance 


1 






Math96<- 1997 Performance 


1 






Math97<- 1997 Performance 


1 






Math93<- Rate of Change 


4 






Math94<- Rate of Change 


3 






Math95<- Rate of Change 


1.392 


.339 


4.103 


Math96<- Rate of Change 


1 






Math97<- Rate of Change 


0 






Latent Variable Means: 








1997 Performance 


521.612 


2.472 


211.041 


Rate of Change 


-2.701 


.264 


-10217 


Variances/Covariances: 








1997 Perform-Rate of Change 


-1.747 


6.128 


-0285 


1997 Performance 


496.663 


79.441 


6252 


Rate of Change 


2.432 


1.039 


2342 


el 


31.536 


92 73 


3.401 


e2 


32.700 


6.963 


4.696 


e3 


71.019 


12.235 


5.805 


e4 


23.254 


5.746 


4.047 


eS 


47.284 


10.393 


4.550 



It should be noted that a non-linear rate of change was estimated in the model. The chi-square 
difference between a model assuming linear change and the non linear rate of change model described in 
Table VII. 1 was 3.11 with 1 df (p<.10) and the RMSEA (root mean square error of approximation, 
Browne and Cudeck, (1993) was reduced by .03. Therefore, a non-linear rate of change in the Level 1 
model was assumed. 

The structural component of the structural equation model is used to reflect factors which are 
hypothesized to explain the variability in 1997 MSPAP performance (intercepts) and rates of change 
(slopes): q = a + (iq + Q where, q is defined as above, a is a vector of population means for the latent 
variables, P is a matrix of structural slopes for the effects among endogenous and exogenous r| variables 
(e.g., variables included to explain individual differences in intercepts and slopes), and C, are structural 
residuals. In Figure VII.3, two dimensions from the teacher questionnaire and die variable, percent free 
or reduced lunch were introduced into the growth model and paths are included from these variables to 
the latent variables (1997 MSPAP performance and rate of change). The structural residuals are given by 
dl and d2 in the figure, and the relationship between 1997 MSPAP performance and rate of change is 
estimated through these two residual parameters. As indicated previously, only two dimensions were 
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introduced since the school sample size is relatively small (n=86). Also, in theory, it would be possible 
to incorporate the confirmatory factor analysis model for the teacher questionnaire directly with the 
growth model rather than use die composite variables for the two dimensions. However, given the 
sample size in the present study, such a model was overly complex to be estimated. 
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Table VII .2 presents tbe regression coefficients for the variables introduced to explain variation in 
1997 MSPAP performance and changes in performance over time. The chi-square statistic for model- 
data-fit was 24.989 with 18 df (p=.125) indicating that the null hypothesis that the variance-covariance 
matrix implied by the model in Table VII .2 equals the observed variance-covariance matrix could not be 
rejected. The RMSEA statistic was .068 which is within the acceptable range (Browne and Cudeck, 
1993). As can be seen, the variable Percent Free Lunch is significantly related to 1997 MSPAP 
performance. Thus, increases in the percentage of students receiving free or reduced lunch is associated 

| 

with lower levels of MSPAP performance in 1997. The only factor that was found to significantly 
explain variability in rates of change was the teacher questionnaire dimension, MSPAP Impact This 

i indicates that higher levels of teacher reports of MSPAP having a direct impact on instruction are 

associated with greater rates of decrease in performance from. 1997 to 1993 or higher levels of rate of 
change in MSPAP school performance. Finally, it is interesting to note that, although increases in the 
percentage of students receiving free lunch is- associated with lower levels of MSPAP performance in 
1997, corresponding increases were not significantly associated with rate of change in MSPAP 
performance over time. 

I It is important to note that the school sample size for this analysis was relatively small (n=86) and 

therefore, the results should be interpreted cautiously and additional studies should be conducted. Data 

t 

; that was collected in die science and social studies areas were based on a larger sample size. Therefore, 

growth model analyses using these data sets may provide more stable results than the analyses from the 

1 mathematics data set 



Table VII .2 Results for the Level 2 Growth Model - Factors Introduced to Explain MSPAP 1997 

Performance and Rate of Change 



Measure and Variable 


Estimates 


SE 


t 


Regression Coefficients 








Effects on 1997 Perform. 








Current Math Inst. 


6.929 


5.666 


1.223 


MSPAP Impact 


.389 


4.164 


.093 


Percent Free Lunch 


-.777 


.059 


-13.134 


Effects on Rate of Change 








Current Math Inst. 


1.214 


1.023 


1.265 


MSPAP Impact 


-1.580 


.752 


-2.101 


Percent Free Lunch 


-.008 


.011 


-.725 
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Vm. Descriptive Results from Mathematics Classroom Activities 



Results from the mathematics classroom activities are discussed in three major sections: 1) 

description of classes and teachers, 2) results from the analysis of the instruction, assessm ent, and 
MSPAP test preparation activities, and 3) results from the analysis of the classroom scoring schemes. 
The first section describes die type and heterogeneity of the math classes from which the activities were 
selected, the sample sizes, the sources of the activities, and rater agreement in coding die activities. In 
the results section for instruction, assessment, and test preparation activities, information is provided 
regarding the format of the activities, the extent to which they reflect the Maryland Learning Outcomes, 
other task features of the activities (e.g., response type required of student, integration with other subject 
areas, etc.), and their similarity to MSPAP-like tasks. Finally, in the third section, results are provided 
for the classroom scoring schemes such as the format of the scoring scheme and their evaluation 
components. The Maryland Learning Outcomes and the format and content of MSPAP tasks served as 
the basis for the coding schemes that were developed for the analysis of the classroom instruction, 
assessment, and MSPAP test preparation activities and the scoring schemes. 

Description of Classes and Teachers who Collected Classroom Activities 

Mathematics teachers were asked to send in approximately 5 to 6 instruction activities, 5 to 6 
assessment activities, and 1 sample of a scoring scheme used from September to December 1996. 
Similarly, in the spring they were asked to send in another set of 5 to 6 instruction activities, 5 to 6 
assessment activities, and 1 sample of a scoring scheme used from January to June 1997. In addition, 
they were also asked to send a sample of a MSPAP test preparation activity used prior to the 
administration of MSPAP. If the teacher taught more than one mathematics class, they were requested to 
obtain these materials from a typical class that they taught A total of 250 mathematics teachers sent in a 
sample of their mathematics classroom activities used during the 1996-97 school year. 

Type of Mathematics Class 

Teachers were asked to indicate the type of math class from which their sample of classroom 
activities was selected. As indicated in Table VIII. 1 nearly all the elementary classes were “general 
math” classes (98%), while only 42% of the middle school classes were “general math” classes. The 
remaining middle school classes were either prealgebra classes (39%) or algebra classes (1 5%). 
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Table VIII. 1 Type of Math Class from ■which the Classroom Materials were Selected 





All Grades 
n=266* 


Elementary 

n=162 


Middle 

n=104 


General Math 


76% 


98% 


42% 


Pre-Algebra 


15% 


0% 


39% 


Algebra, Algebra I, and 
Algebra II 


5% 


0% 


15% 


Not Indicated 


3% 


2% 


4% 



* 16 teachers had a change in the type of math class taught from fall to spring. 



Heterogeneity of Mathematics Class 

Teachers were also asked to indicate the heterogeneity of the students in the math class from which 
their sample of classroom activities were selected. As indicated in Table VIII .2 almost 60% of the 
elementary classes compared to only 25% of the middle school classes were classified by the teacher as 
heterogeneous. A larger percentage of middle school classes (64%) than elementary classes (33%) were 
classified as homogenous, either on-grade, above-grade, or below-grade level. Only a very small 
percentage of die classes were exclusively special education or gifted and talented. 



Table VIII.2 Heterogeneity of Math Class from which the Classroom Materials were Selected 





All Grades 
n=286* 


Elementary 

n=183 


Middle 

n=103 


Heterogeneous 


47% 


59% 


25% 


Homogenous 


44% 


33% 


64% 


On-Grade 


24% 


19% 


33% 


Above-Grade 


11% 


8% 


15% 


Below-Grade 


9% 


6% 


16% 


Exclusively Special 
Education 


6% 


6% 


6% 


Exclusively 

Gifted/Talented 


1% 


1% 


3% 


Not Indicated 


2% 


2% 


3% 



* 36 teachers had a change in the level of math class taught from fall to spring. 



Sample Sizes for Teachers and Classroom Activities by Grade Level 

On average across the entire school year, approximately 16 classroom instruction and assessment 
activities were collected per teacher. In die fall, 236 mathematics teachers sent in 10 classroom activities 
on average, and in the spring, 163 mathematics teachers sent in 10 classroom activities on average. For 
each grade level, Table VIII.3 indicates the number and percentage of teachers who sent in classroom 
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activities and also the total number and percentage of all classroom activities received. For example, 39 
2 nd grade teachers sent in a total of 591 classroom activities. The percentages across grades for the 
number of teachers and the number of activities are somewhat similar, although a sligh tly smaller 
percentage of off-grade teachers (2, 4, and 7) than on-grade teachers (3, 5, and 8) sent in classroom 
activities. 



Table VIII.3 Number of Teachers and Classroom Activities bv Grade Level 



Grade 


Teachers 


Activities 


Number 


Percentage 


Number 


Percentage 


2 


39 


15% 


591 


15% 


3 


49 


19% 


854 


22% 


4 


31 


12% 


454 


11% 


5 


j 45 


18% 


698 


18% 


7 


37 


15% 


639 


16% 


8 


52 


21% 


712 


18% 


Total 


253* 


100% 


3948 


100% 



* 3 teachers changed the grade taught from fall to spring. 



Sample Sizes by Type of Classroom Activity 

Teachers were provided with labels to attach to each activity indicating the type of activity (e.g., 
instruction, assessment, test preparation, scoring scheme). Table VHI.4 shows the number and 
percentage of activities for each type. Across all grades (2, 3, 4, 5, 7, and 8), there was a total of 1940 
instruction activities, 1388 assessment activities, and 332 scoring schemes. For grades 3, 5, and 8 there 
was a total of 125 MSPAP test preparation activities. The table also includes a category called “not 
coded”. These were activities that were not coded for one of two reasons. One reason for not coding an 
activity was because it pertained strictly to another content area such as social studies or science. 
Another reason an activity was not coded was because it consisted only of teacher notes or general lesson 
plans and it was difficult to discern what the students were required to do. 

The percentages across grade levels for each of the types of activities were somewhat similar, 
although slightly more on-grade teachers than off-grade teachers sent in classroom activities. (A table 
indicating the number and percentage of activities by grade level is included in Appendix A.) 
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Table Vin.4 Type of Classroom Activity 





Activities 


Teachers 


Mean Number of 
Activities 
Per Teacher 


Number 


Percentage 


Number 


Percentage 


Instruction 


1940 


49% 


245 


98% 


7.92 


Assessment 


1388 


35% 


214 


86% 


6.49 


MSPAP Test 
Preparation (3, 5,8) 


125 


3% 


51 


35% 


2.45 


Scoring Schemes 


332 


8% 


141 


56% 


2.35 


Not Coded 


163 


4% 


90 


36% 


1.81 



Sources of Classroom Activities 

Teachers were also asked to indicate the source of each activity. For each type of classroom activity, 
Table Vm.S indicates across all grades the number and percentage of activities from a variety of sources. 
The results across grades were quite similar. Over half of the instruction activities (57%) were from 
textbook or commercial resources and 25% were teacher-developed. Approximately equal percentages 
of die assessment activities were from textbook/commercial resources or were teacher-developed (36% 
and 38% respectively). While the percentage of activities that were county-developed was quite small 
for instruction and assessment activities, there was a slightly larger percentage of assessment activities 
(15%) than instruction activities (8%) that were county developed. The percentage of instruction and 
assessment activities obtained from state-level materials, such as MSPAP Release Tasks, Maryland 
Consortium Tasks, and Maryland Performance-Based Exemplars was very small. 

When examining the MSPAP test preparation activities, the sources were somewhat different than 
for the instruction or assessment activities. The percentage of teacher-developed activities was similar 
(33%), however, there was a larger percentage that were county-developed (26%), MSPAP Release 
Tasks (4%) and other state-level materials (10%). The sources for die scoring schemes were similar to 
the test preparation activities. About 35% were teacher developed, 20% were county-developed, and 9% 
were state-level materials. 



Table Vm.5 Sources of Classroom Activity 





Instruction 
n=1948 ' 


Assessment 

d=1388 


MSPAP Test 
Prep 

( 3 rd, 5 th, gth) 
n=125 


Scoring 

Scheme 

n=332 


Teacher/Other Teacher/ 
School Developed 


25% 


36% 


33% 


35% 


Textbook/Commercial 

Resources 


57% 


38% 


15% 


15% 


County/Another County 
Developed 


8 % 


15% 


26% 


20 % 


Teacher and Textbook/ 
Teacher and County 
Developed 


4% 


5% 


4% 


2 % 


MSPAP Release Tasks 


< 1 % 


1 % 


4% 


2 % 


MD Consortium/ Exemplars 


1 % 


1 % 


10 % 


9% 


Other/ Teacher and 
Student/Class (Scoring) 


< 1 % 


< 1 % 


1 % 


< 1 % 


Cannot Be Determined 


5% 


4% 


9% 


15% 



Rater Agreement 

A total of four raters coded the classroom activities. A formal training session was conducted to 
familiarize the raters with the coding scheme using a sample set of pre-coded activities. Then, the raters 
coded another set of sample activities independently and their codes were compared and discussed by the 
group. After the formal training was complete, pairs of raters individually coded a set of classroom 
activities from a school (elementary or middle) for a certain collection period (fall or spring). The pair 
of raters met to discuss their discrepancies and reached a consensus on the codes for each activity. This 
was done to ensure that all raters shared a common understanding of the coding scheme. Thus, for a 
small percentage of classroom activities (7%), one set of codes, agreed upon by two raters, was obtained. 

After it was determined that the raters reached a shared unders tanding of the coding scheme and 
were proficient in applying it to a variety of classroom activities, each rater individually coded sets of 
classroom activities. Approximately 21% of the sets of classroom activities (an elementary or middle 
school teacher’s activities from either fall or spring) were coded individually by two raters. The overall 
adjusted rate of agreement between the raters was then calculated^. See Appendix B for a description of 
how the adjusted rate of agreement was calculated.) The adjusted rate of agreement was found to be 

2 Percent agreement was considered to be too lenient of an index of rater agreement because for a number of the categories to be 
coded there were a range of options that could be selected. As an example, for the content learning outcome, one to eight 
content outcomes could be selected for an activity. However, the majority of the activities had between one to three content 
outcomes coded. A simple percent agreement based on each of die eight outcomes would have inflated the index for rater 

(footnote continued on next page) 
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84% for the instruction, assessment, and test preparation activities and 81% for the scoring schemes. In 
addition to examining die agreement between rater pairs, the accuracy of raters’ codes was examined for 
23% of the sets of classroom activities. This was accomplished by comparing a rater’s set of codes with 
a “valid” set of codes obtained by die lead rater who had been involved in die conceptualization and 
development of the coding scheme. The adjusted rate of agreement was 87% for die instruction, 
assessment, and test preparation activities, and 74% for the scoring schemes. 

Descriptive Results from Instruction, Assessment, and 
MSPAP Test Preparation Classroom Activities 

Classroom Activity Format 

Each of the MSPAP tasks consists of a series of items that are related to the same problem context or 
situation. Therefore, for the classroom instruction, assessment, and test preparation activities, if all die 
items within an activity were related to die same problem context or situation (e.g., all hems were related 
to designing rooms in a house), the activity was treated as one task. Whereas, if each item was not 
related to the same context, the activity was treated as distinct items (e.g., a set of traditional text-book 
like word problems). Table VIII. 6 indicates die percentage of instruction, assessment, and test 
preparation activities that were considered as a task with related items, as a set of distinct items, and as a 
sole item. This information is provided for each grade and across grades. In some cases, an activity 
could be coded as both consisting of a task with several items related to the same context and a set of 
distinct hems. As might be expected, this occurred more often for the classroom assessment activities 
and the MSPAP test preparation activities as compared to the instruction activities. 



agreement Thus, an adjusted percent agreement was used. The procedure for obtaining the adjustment is provided in Appendix 
B. 



Table VHI .6 Activitv Format for Mathematics Classroom Activities 



Instruction 


All 

grades 


2 nd 


3 rd 


4 th 


~5=“ 


7 th 


pi~ 


Task with several items 
related to the same context 


47% 


38% 


48% 


49% 


53% 


47% 


48% 


Distinct items 


54% 


57% 


50% 


56% 


53% 


55% 


54% 


Only one item 


7% 


13% 


8 % 


3% 


6 % 


3% 


6 % 


Assessment 


One task with several items 
related to the same context 


38% 


32% 


49% 


29% 


47% 


30% 


36% 


Distinct items 


71% 


72% 


60% 


81% 


66 % 


83% 


70% 


Only one item 


7% 


11 % 


6 % 


6 % 


6 % 


8 % 


7% 


MSPAP Test Preparation (3 rd , 5 th , 1 


t Ul) 


One task with several items 
related to the same context 


85% 




90% 




79% 




85% 


Distinct items 


16% 


— 


10 % 


— 


14% 


— 


21 % 


Only one item 


6 % 


— 


3% 


— 


18% 




2 % 



All Grades . Across grades, 47% of the instruction activities consisted of a task with several items 
which were related to the same problem context or situation and 54% of the activities consisted of a set 
of distinct items. A larger percentage of assessment activities (71%), as compared to instruction 
activities (54%), consisted of a set of distinct items. In contrast, a very small percentage of MSPAP test 
preparation activities (16%) consisted of a set of distinct items and most of the MSPAP test preparation 
activities (85%) consisted of a task with several items related to the same context. As mi gh t be expected, 
the MSPAP test preparation activities tended to be more similar in form to the actual MSPAP taslrg^ as 
compared to the classroom instruction and assessment activities. 

Differences Across Grades. In general, differences across grades were small. Of particular interest, 
a larger percentage of instruction and assessment activities from the elementary on-grades ( 3 ^ and 5 ^), 
as compared to the off-grades ( 2 n< ^ and 4^*), consisted of a task with several items related to the same 
context. This is particularly the case for the assessment activities. 



Maryland Learning Process Outcomes 

The classroom instruction, assessment, and MSPAP test preparation activities were coded in terms of 
whether they focused on problem solving and/or reasoning as defined by the MLO’s and MSPAP. As 
indicated previously, if all the items within an activity were related to the same problem context or 
situation, the activity was treated as one task. If each item was not related to the same context, the 
activity was treated as distinct items. Thus, when die activity was considered as one task, it was coded as 
either focusing on problem solving and/or reasoning, or not Whereas, if the activity was coded as 
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distinct items, it could be coded into (me of three categories: ( 1 ) all items with a focus on problem 
solving and/or reasoning, (2) only some items with such a focus, or (3) no items with such a focus. The 
table below shows the results for each grade and across all grades. 

1 All Grades . As indicated in Table Vm.7, across the grades, 52% of die classroom instruction 

; activities required problem solving and reasoning to some extent, with 44% of these activities having a 

predominate focus on problem solving and/or reasoning. Across die grades, 46% of die classroom 
assessment activities required problem solving and/or reasoning to some extent, with 29% of these 
1 having a predominate focus on problem solving and/or reasoning. As might be expected a larger 

percentage of MSPAP test preparation activities used in the 3^, 5^, and 8 *h grade classes (91%), as 
compared to the instruction and assessment activities, required problem solving and/or reasoning to some 
extent Moreover, 84% of the MSPAP test preparation activities primarily focused on problem solving 
and/or reasoning. 



Table VH1.7 Process Learning Outcomes in Mathematics Classroom Activities 



Instruction 


All Grades 


2 nd 


3 rd 


4® 


5 th 


7 th 


8 th 


Problem solving and/or reasoning 


44% 


39% 


47% 


42% 


47% 


44% 


43% 


Some items with and some items 
without problem solving and/or 
reasoning 


8 % 


8 % 


7% 


10 % 


12 % 


6 % 


8 % 


No problem solving nor reasoning 


48% 


53% 


46% 


49% 


41% 


51% 


49% 


Assessment 
















Problem solving and/or reasoning 


29% 


26% 


43% 


21 % 


34% 


18% 


28% 


Some items with and some items 
without problem solving and/or 
reasoning 


17% 


16% 


15% 


20 % 


18% 


23% 


12 % 


No problem solving nor reasoning 


54% 


58% 


42% 


59% 


48% 


59% 


60% 


MSPAP Test Preparation 
















Problem solving and/or reasoning 


84% 


— 


90% 


— 


86 % 


— 


79% 


Some items with and some items 
without problem solving and/or 
reasoning 


7% 




3% 




11 % 




9% 


No problem solving nor reasoning 


9% 


— 


8 % 


— 


4% 


— 


12 % 



Differences Across Grades . In general, the 3*^ and 5& grade (i.e., elementary on-grades) 
instruction and assessment activities focused on problem solving and/or reasoning more often than the 
activities in the other grade levels. This is particularly the case for the assessment activities. 

Relationship Between Classroom Activities and Questionnaire Responses . The results above are 
fairly consistent with the results from the student and teacher questionnaires. Overall, 54 % of the 
students indicated that tasks that involved problem solving were used in their classrooms on a daily or 
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weekly basis, with the elementary grades (3^ and 5^) having a larger percentage than the middle grades 
(7th and 8th). Depending on grade level, between 67% and 89% of the teachers indicated that they use 
tasks that emphasize problem solving and reasoning on a daily (between 25% and 45%) or weekly 
(between 42% and 48%) basis. A larger percentage of 3 rd and 5th grade teachers reported that on a dail y 
or weekly basis they emphasized problem solving and reasoning (89%), as compared to 2 nd and 4th 
(83%), 8th (74%), and 7th ( 67 %) grade teachers. 

Further, there was a larger percentage of 3^ d and 5th grade teachers, as compared to the other grade- 
level teachers, who indicated that both MSPAP and the MLO’s had a “great” impact on their classroom 
activities and that the impact increased a “great” extent since 1992. When asked specifically about the 
impact of MSPAP on their instruction and assessment, a larger percentage of 3 rd and 5* grade teachers 
indicated that MSPAP had a moderate or great amount of positive impact on instruction and assessment 

Maryland Learning Content Outcomes 

The classroom instruction, assessment, and MSPAP test preparation tasks were also coded in terms 
of their content emphasis. The Maryland Learning Content Outcomes provided the basis for the 
classification and are provided in the table below. It should be noted that the ‘prealgebra’ category was 
extracted from Maryland’s ‘algebra’ category. For the purpose of this analysis, ‘prealgebra’ included 
the evaluation of expressions and order of operations, and ‘algebra’ was restricted to working with 
unknowns. All other categories were defined according to the Maryland Learning Outcomes. 

All, Grades. Each activity, regardless if it was one task or a set of distinct items, could be coded for 
more than one content outcome. Thus, a content outcome had one or more opportunities to be selected 
for an activity. Of the 1,940 instruction activities, 80% (1,550) were coded solely for one content 
outcome, 17% were coded for two content outcomes, and 3% were coded for three content outcomes. 
The table below indicates die percentage of times across all grade levels that an activity was coded for 
each of the content outcomes when one, two, and three content outcomes were coded per activity. 
(Similar tables of results for each grade level are provided in Appendix C.) The majority of instruction, 
assessment, and MSPAP test preparation activities were coded for only one content outcome. As might 
be expected, however, a larger percentage of assessment and test preparation activities (34%), as 
compared to instruction activities (20%), were coded for more than one content outcome. It should be 
noted that 3 A> of the assessment activities and 4% of the test preparation activities were coded for four or 
more content outcomes. 

The last column in the table, labeled ‘overall’, indicates the percentage of times each of the content 
outcomes were coded for an activity regardless if one, two, or three content outcomes were coded per 
activity. For example, for those instruction activities in which only one content outcome was selected, 
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numbers and operations was selected for 54% of the activities and it was not selected for 46% of those 
activities; for those activities in which two content outcomes were selected, numbers and operations was 
selected for 74% of the activities and it was not selected for 26% of those activities; and for those tasks 
in which three content outcomes were selected, numbers and operations was selected for 92% of the 
activities and it was not selected for 8% of those activities. Lastly, regardless of whether one, two, or 
three content outcomes were selected, numbers and operations was selected for 59% of the activities and 
it was not selected for tiie 41% of them. 

As suggested in Table VIII.8, numbers and operations was emphasized more often than the other 
content outcomes in the classroom, instruction, assessment and test preparation activities. In general, 
geometry/measurement was the next most emphasized content outcome, with statistics being the third 
most emphasized content outcome. For example, overall, 59% of the instruction tasks were coded for 
numbers and operations, 27% of the instruction tasks were coded for geometry/measurement, and 19% 
were coded for statistics. It should be noted that although numbers and operations was the most 
frequently occurring content outcome selected for MSPAP test preparation activities, overall, the content 
outcomes of geometry/measurement, statistics, probability and pattems/relationships were selected more 
often for test preparation activities than for die instruction and assessment activities. Overall, 49% of the 
test preparation tasks were coded for numbers and operations, 39% of the test preparation activities were 
coded for geometry/measurement, and 29% were coded for statistics^. 



3 When more than one content outcome was selected, the content area that was emphasized primarily was indicated. However, if 
there was an equal emphasis in the content areas that were selected, it was indicated as such. For those tacVc in which more than 
one content area was selected, the results were consistent with those in the table, in that, numbers and operations was chosen 
most often as the primary emphasis, followed by geometry/measurement and then, statistics. 



Table VIII.8 Content Learning Outcomes for Mathematics Classroom Activities - All 





Number Of Learning Outcomes Selected 


Overall 


One 


Two 


Three 


Instruction 


n= 1,550 (80%) 


n=320 (17%) 


n=49 (3%) 




Numbers and Operations 


54% 


74% 


92% 


59% 


Geometry/ Measurement 


21% 


48% 


74% 


27% 


Statistics 


14% 


34% 


43% 


19% 


Probability 


5% 


15% 


20% 


7% 


Pattern and Relationships 


2% 


10% 


25% 


5% 


Prealgebra 


1% 


4% 


14% 


2% 


Algebra 


3% 


16% 


33% 


6% 


None 


1% 


0% 


0% 


<1% 


Assessment 


n=916 (66%) 


n=317 (23%) 


n=117 (8%) 




Numbers and Operations 


58% 


80% 


96% 


67% 


Geometry/ Measurement 


19% 


45% 


j 58% 


30% 


Statistics 


12% 


27% 


35% 


20% 


Probability 


3% 


6% 


15% 


6% 


Pattern and Relationships 


1% 


10% 


23% 


6% 


Prealgebra 


1% 


8% 


25% 


6% 


Algebra 


4% 


25% 


48% 


14% 


None 


1% 


0% 


0% 


1% 


MSPAP Test Preparation 


n=80 (66%) 


n=28 (23%) 


n=13 (11%) 




Numbers and Operations 


38% 


61% 


77% 


49% 


Geometry/ Measurement 


24% 


57% 


77% 


39% 


Statistics 


19% 


39% 


54% 


29% 


Probability 


11% 


21% 


38% 


18% 


Pattern and Relationships 


1% 


14% 


38% 


10% 


Prealgebra 


r_ 0% 


0% 


0% 


2% 


Algebra 


6% 


7% 


15% 


7% 


None 


1% 


0% 


0% 


1% 



Relationship Be tween Classroom Activities and Questionnaire Responses . The order of emphasis 
for the content outcomes is fairly consistent with what was reported in the teacher questionnaires. The 
majority of the teachers indicated in the questionnaires that they emphasized numbers and operations the 
most, followed by geometry/measurement, pattems/algebra, statistics, and probability. 

Differences Acros s Grades . Table Vm.9 shows the overall percentages for each grade level. The 
overall percentages reflect the percentage of times each of the content outcomes was coded for an 
activity regardless if one, two, or three content outcomes were coded per activity. As might be expected, 
none of the elementary instruction, assessment, and test preparation activities were coded for algebra and 
only a very small percentage were coded for prealgebra. Although prealgebra and algebra were more 
predominant in the middle school grades, especially 8^* grade, numbers and operations was the most 
often coded content outcome for the middle school activities. However, in general, the percentage of 
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activities that were coded for numbers and operations decreased across grades, from the 2°d grade to the 
8th grade. 



Table VIII.9 Content Learning Outcomes for Mathematics Classroom Activities- For Each Grade 



Instruction 


All 

grades 


2^3 


3rd 


4th 




yth 


§fE 


Numbers and Operations 


59% 


65% 


63% 


66% 


58% 


54% 


45% 


Geometry/ Measurement 


27% 


22% 


27% 


26% 


30% 


28% 


31% 


Statistics 


19% 


18% 


18% 


15% 


24% 


18% 


19% 


Probability 


7% 


6% 


8% 


7% 


9% 


9% 


5% 


Pattern and Relationships 


5% 


7% 


4% 


7% 


5% 


2% 


4% 


Pre- Algebra 


2% 


0% 


0% 


<1% 


1% 


4% 


7% 


Algebra 


6% 


0% 


0% 


1% 


1% 


16% 


19% 


None 


<1% 


1% 


<1% 


<1% 


0% 


<1% 


1% 


Assessment 


Numbers and Operations 


67% 


77% 


69% 


76% 


68% 


68% 


50% 


Geometry/ Measurement 


30% 


24% 


30% 


32% 


40% 


29% 


26% 


Statistics 


20% 


17% 


22% 


14% 


25% 


17% 


20% 


Probability 


6% 


4% 


5% 


3% 


9% 


5% 


6% 


Pattern and Relationships 


6% 


10% 


9% 


4% 


8% 


4% 


2% 


Pre-Algebra 


6% 


0% 


<1% 


0% 


1% 


17% 


14% 


Algebra 


14% 


1% 


<1% 


1% 


2% 


37% 


40% 


None 


1% 


<1% 


1% 


W 


2% 


1% 


1% 


MSPAP Test Preparation 


Numbers and Operations 


49% 


— 


59% 


— 


54% 




40% 


Geometry/ Measurement 


39% 


— 


49% 


- 


43% 


— 


31% 


Statistics 


29% 




26% 


- 


32% 




29% 


Probability 


18% 


— 


5% 




21% 




24% 


Pattern and Relationships 


10% 




21% 


— 


11% 


— 


2% 


Pre- Algebra 


2% 


— 


0% 




7% 


— 


0% 


Algebra 


7% 


— 


0% 


— 


0% 


— 


16% 


None 


1% 


— 


0% 




4% 


— 


0% 



Other Task Features 

In addition to the process and content outcomes, other features of the classroom activities were 
coded including: interpretation of graphs, tables, or charts; integration with other subject areas; use of 
realistic contexts; use of manipulatives, calculators, and computers; and the type of response required by 
students. The analysis of instruction, assessment, and test preparation activities by these features provide 
additional information for describing their similarity to MSPAP tasks. 

Interpretation of Graphs. Tables, or Charts . MSPAP tasks that assess mathematical problem 
solving and reasoning often require students to interpret information represented in graphs, tables, or 
charts. Table Vm.10 indicates the percentage of classroom instruction, assessment, and test preparation 



activities that required students to interpret information in graphs, tables, or charts. Across the grades, 
27% of the instruction activities, 30% of the assessment activities, and 56% of the MSPAP test 
preparation activities required students to interpret information in graphs, tables, or charts. 



Table VII I. 10 Inte rpreting Graphs. Tables, or Charts within Mathematics Classroom Activities 



Instruction 


All Grades 


2 nd 


3 rd 


4 th 


5 th 


7 th 


8 t& 


Yes 


27% 


21 % 


23% 


27% 


30% 


31% 


33% 


No 


73% 


79% 


78% 


73% 


70% 


69% 


67% 


Assessment 
















Yes 


30% 


28% 


36% 


20 % 


39% 


25% 


27% 


No 


70% 


72% 


65% 


80% 


61% 


75% 


73% 


MSPAP Test Preparation 
















Yes 


56% 


— 


72% 


— 


43% 


— 


52% 


No 


44% 


— 


28% 


— 


57% 


— 


48% 



Differences Across Grades . The percentage of instruction activities that required students to 
interpret information in graphs, tables or charts was similar across die grades. However, a slightly larger 
percentage of 3 rc * and 5^ grade assessment activities and a relatively larger percentage of 3 *^ grade 
MSPAP test preparation activities required interpreting information from graphs, tables, or charts. 

Integra tion with Other Subject Areas . The majority of die MSPAP tasks that assess mathematics 
are integrated with other subject areas including, science, social studies, reading, writing, and language 
usage. Therefore, the mathematics classroom activities were analyzed in terms of whether they were 
integrated with other subject areas. The integration categories include science, social studies, reading, 
writing, art, and glyphs/tesselations. An activity was coded for reading if the student was required to 
read a story or other type of reading material prior to solving die mathematics problem. Integration with 
writing was selected if the activity required students to provide short explanations or long explanations, 
to create or pose mathematics problems, or to write a letter or persuasive paragraph based on the 
mathematics problem. Art was selected if the student was asked to paint, create a model, or the like. If 
students were required to create a glyph or tessellation this was also coded as a form of integration 
because it required some level of artistic expression; however, it was coded as its own category. 

Each activity, regardless if it was one task or a set of distinct items, could be coded for more than 
one category for integration. Of the 1,940 instruction activities, 13% were coded for more than one 
integration category. Of the 1388 assessment activities, 10% were coded for more than one integration 
category. Whereas, 38% of the MSPAP test preparation activities that were coded for more than one 
integration category. Table VIII. 11 indicates the percentage of times each category of integration was 
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coded for an activity when one or more integration categories were coded. The table also indicates die 
percentage of times the activities required no integration. 

Across grades, 40% of the mathematics instruction activities and 33% of the assessment activities 
were integrated with other subject areas, while 82% of the test preparation activities were integrated with 
other subject areas. The most common forms of integration regardless of the type of activity were within 
the writing area. Overall, 36% of the instruction activities, 30% of the assessment activities, and 78% of 
the MSPAP test preparation activities involved writing. As an example, for the instruction activities, 
27% were coded for short explanations, 10% were coded for long explanations, 5% were coded for 
posing mathematical problems, and 5% were coded for writing a letter or persuasive paragraph. It should 
be noted again that an activity could be coded for more than one type of writing. Although differences 
across grades were rather small, there were slightly higher percentages of activities requiring long 
explanations or writing letters/persuasive paragraphs in the middle school grades than the elementary 
school grades, especially for the assessment activities. 



Table Vin.l 1 Integration of other Subject Areas in Mathematics Classroom Activities 



Instruction 


All Grades 


2 nd 


3 rd 


4 th 


I®" 


7 tn 


gtn 


Science 


4% 


3% 


3% 


2 % 


6 % 


6 % 


4% 


Social Studies 


2 % 


1 % 


1 % 


2 % 


3% 


2 % 


2 % 


Reading Materials 


2 % 


1 % 


4% 


2 % 


1 % 


1 % 


3% 


Writing 

Short Explanations 


27% 


18% 


28% 


27% 


31% 


28% 


29% 


Long Explanations 


10 % 


7% 


9% 


7% 


10 % 


11 % 


14% 


Creating/Posing Math Problems 


5% 


4% 


7% 


6 % 


6 % 


5% 


2 % 


Letters/Persuasive Paragraphs, etc. 


5% 


4% 


5% 


5% 


4% 


7% 


6 % 


Language Usage 


1 % 


1 % 


1 % 


0 % 


0 % 


1 % 


0 % 


Art 


1 % 


0 % 


1 % 


0 % 


1 % 


2 % 


2 % 


G lyphs/T esselations 


2 % 


3% 


5% 


2 % 


2 % 


0 % 


1 % 


No Integration 


60% 


58% 


41% 


51% 


40% 


41% 


41% 


Assessment 
















Science 


2 % 


3% 


2 % 


3% 


3% 


1 % 


1 % 


Social Studies 


1 % 


1 % 


1 % 


3% 


1 % 


0 % 


1 % 


Reading Materials 


1 % 


0 % 


1 % 


0 % 


0 % 


0 % 


2 % 


Writing 

Short Explanations 


23% 


16% 


37% 


16% 


25% 


20 % 


21 % 


Long Explanations 


9% 


5% 


7% 


3% 


8 % 


10 % 


18% 


Creating/Posing Math Problems 


3% 


3% 


3% 


3% 


3% 


3% 


! 1 % 


Letters/Persuasive Paragraphs, etc. 


5% 


1 % 


5% 


2 % 


5% 


8 % 


9% 


Language Usage 


1 % 


1 % 


0 % 


1 % 


0 % 


2 % 


1 % 


Art 


1 % 


0 % 


0 % 


0 % 


2 % 


1 % 


1 % 


Glyphs/Tesselations 


1 % 


0 % 


2 % 


1 % 


2 % 


0 % 


1 % 


No Integration 


67% 


69% 


43% 


71% 


53% 


57% 


47% 


MSPAP Test Preparation 
















Science 


10 % 




21 % 


— 


4% 


— 


5% 


Social Studies 


6 % 


- 


10 % 


— 


4% 


— 


5% 


Reading Materials 


2 % 


- 


5% 


— 


4% 


— 


0 % 


Writing 

Short Explanations 


65% 




69% 


__ 


71% 




59% 


Long Explanations 


34% 




23% 


— 


32% 


— 


41% 


Creating/Posing Math Problems 


2 % 




0 % 


— 


4% 


— 


3% 


Letters/Persuasive Paragraphs, etc. 


17% 


- 


10 % 


— 


32% 


— 


14% 


Language Usage 


2 % 


- 


5% 


— 


0 % 


— 


0 % 


Art 


2 % 


- 


3% 


— 


0 % 


— 


2 % 


Glyphs/Tesselations 


2 % 


- 


8 % 


— 


0 % 


— 


0 % 


No Integration 


18% 


- 


0 % 


— 


0 % 


- 


0 % 



Relationship Between Classroom Activities and Questionnaire Responses . Overall, 50% of the 
teachers reported on the questionnaire that they use integrated tasks on a daily or weekly basis which is 
fairly consistent with the results for the classroom activities. However, only 26% of the students 
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indicated that they work on integrated tasks on a daily or weekly basis. It may be die case that students 
do not perceive tasks that require writing as being integrated with another subject area. 

Use of a Realistic Context . In general, MSPAP mathematics tasks are set in a realistic context; 
therefore, die classroom mathematics activities were analyzed in terms of whether they pertained to real- 
life situations, or not As indicated previously, if all items within an activity were related to the same 
context or situation, the activity was treated as one task. If each item was not related to the same context 
the activity was treated as distinct items. Thus, when the activity was considered as one task, it was 
coded as either embedded in a realistic context or not. Whereas, if the activity was coded as distinct 
items, it could be coded into three categories: ( 1 ) all items embedded in a realistic context ( 2 ) only some 
items were embedded in a realistic context, or (3) no items were embedded in a realistic context 

As indicated in Table Vm.12, across grades, 50% of the instruction activities and 58% of the 
assessment activities were embedded in a realistic context to some extent A higher percentage of 
MSPAP test preparation activities (80%) were embedded in a realistic context to some extent In 
general, differences across grades were small. 



Table VIII. 12 Realistic Context Used in Mathematics Classroom Activities 



Instruction 


All Grades 


2 nd 


3 rd 


4 th 


5® 


7 th 


~w~ 


Yes 


39% 


34% 


39% 


36% 


41% 


45% 


39% 


Yes for some items, no for others 


11 % 


4% 


8 % 


15% 


18% 


9% 


11 % 


No 


50% 


62% 


53% 


49% 


42% 


46% 


50% 


Assessment 
















Yes 


27% 


28% 


37% 


22 % 


27% 


18% 


28% 


Yes for some items, no for others 


31% 


31% 


27% 


36% 


35% 


35% 


25% 


No 


42% 


41% 


36% 


42% 


39% 


47% 


48% 


MSPAP Test Preparation 
















Yes 


74% 


— 


82% 




71% 


— 


71% 


Yes for some items, no for others 


7% 


- 


5% 


— 


4% 




7% 


No 


20 % 


- 


13% 


— 


25% 


— 


22 % 



Relationship Between Classroom Activities and Questionnaire Responses . The results above are 
somewhat consistent with the results from the teacher questionnaire. Overall, 50% of the instruction 
activities and 58% of the assessment activities were embedded in a realistic context, to some extent, and 
78% of the teachers on the questionnaire indicated that they use realistic activities on a daily (32%) or 
weekly (46%) basis. However, only 46% of the students indicated on the questionnaire that they work 
on activities that are set in a realistic context on a daily or weekly basis. 
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Use of Manipulatives. Calcul ators, and Computers. Many of die MSPAP tasks require students to 
use manipulatives. Table Vlll. 13 indicates the percentage of classroom instruction, assessment, and test 
preparation activities that were coded for the use of manipulatives, calculators, and/or computers. It 
should be noted that each activity could be coded for more than one of these categories, although this 
occurred infrequently. For 98% of the instruction and assessment activities, only one category was 
coded. Similar for 97% of the test preparation activities, only one category was coded. As indicated in 
the table, manipulatives were coded for 28% of the instruction activities, 15% of die assessm ent 
activities, and 25% of the MSPAP test preparation activities. Calculators were coded less frequently and 
evidence for the use of computers in the activities was rare. 

The results indicate that the percentage of activities requiring the use of manipulatives, calculators, 
and computers appears to be rather small. However, it is possible that their actual use in the classroom 
activities occurred more frequently than represented here. Manipulatives, calculators, and computers 
were coded when there was an explicit mention of their use somewhere in the activity, and teachers were 
not specifically asked to indicate whether they were used. Therefore, it may be the case that they were 
available for students to use in the classroom even though they were not explicitly written into the 
classroom activity. 

Table Vm.13 Manipulatives. Calculators and Computers Used with Mathematics Classroom Activities 



Instruction 


All Grades 


2 nd 


3 rd 


4 th 


5 th 


7 tn 


8 th 


Manipulatives 


28% 


32% 


32% 


25% 


24% 


27% 


23% 


Calculators 


4% 


1 % 


6 % 


6 % 


4% 


4% 


4% 


Computers 


1% 


0 % 


1 % 


0 % 


0 % 


1 % 


1 % 


None 


69% 


68 % 


64% 


71% 


73% 


70% 


73% 


Assessment 
















Manipulatives 


15% 


24% 


19% 


12 % 


18% 


8 % 


10 % 


Calculators 


3% 


1 % 


4% 


1 % 


3% 


3% 


5% 


Computers 


o 


0 % 


0 % 


1 % 


0 % 


0 % 


0 % 


None 


83% 


76% 


78% 


88 % 


80% 


89% 


88 % 


MSPAP Test Preparation 
















Manipulatives 


25% 


— 


18% 


■ — 


18% 


— 


33% 


Calculators 


15% 


— 


28% 


— 


7% 


— 


10 % 


Computers 


0 % 


— 


0 % 


— 


0 % 


— 


0 % 


None 


63% 




54% 




79% 


— 


62% 



Difference s Across Grades. A slightiy larger percentage of the instruction activities in the 2 n< * and 
3 rd grades, as compared to the other grades, required the use of manipulatives. With regard to the 
assessment activities, elementary grades, and in particular the 2 ®d grade required manipulatives more 
often than the middle grades. It is interesting to note that a very small percentage of the 8 ^ grade 



assessment activities required manipulatives (10%); whereas, the 8*h grade test preparation activities, as 
compared to the other on-grades, wore coded die most for die use of manipulatives (33%). 

Relationship Between Classroom Activities and Questionnaire Responses . The questionnaires asked 
both students and teachers how often their instruction and assessment activities involve manipulatives. 
Overall, 59% of the teachers indicated on die questionnaire that they used instruction activities that 
involve manipulatives on a daily or weekly basis, and 48% indicated that they use manipulatives with 
assessment activities on a weekly basis. Whereas 30% of the students indicated that they work on 
instruction activities that involve manipulatives on a daily or weekly basis, and only 13% indicated that 
they work on assessment activities that involve manipulatives. The student results, as compared to the 
teacher results, appear to be more consistent with the classroom activity results. However, as mentioned 
before teachers may have provided us with classroom activities in which students used manipulatives, 
but it was not apparent when examining the activities. Further, elementary school teachers and students, 
as compared to middle school teachers and students indicated on the questionnaire that they use 
manipulates more often which is fairly consistent with the classroom activity results. 

Response Required of Student The MSPAP tasks that are scored for mathematics require students 
to respond in a variety of ways including providing numerical values, short answers, short and long 
explanations, and graphs. The mathematics classroom activities were coded according to response type 
to examine the extent to which they reflect MSPAP response types. Each classroom activity, regardless 
if it was one task or a set of distinct items, could be coded for more than one response type. Thus, a 
response type had one or more opportunities to be selected for an activity. Across grades, of the 1,940 
instruction activities, 34% (664) were coded solely for one response type, 30% were coded for two 
response types, 18% were coded for three response types, 10% were coded for four response types, and 
6% were coded for five response types. It should be noted that approximately 2% of the activities were 
coded for more than five response types which are not reflected in the table. These percentages are 
similar for the assessment activities, however a larger percentage of test preparation activities were 
coded for more than one response type (88%). Table VIII. 14 indicates the percentage of times an activity 
was coded for each of the response types when one, two, three, four or five response types were coded 
per activity across all grade levels. Tables summarizing results for each grade are provided in Appendix 
D. The last column in foe table, labeled ‘overall’, indicates foe percentage of times an activity was coded 
for each of foe response types regardless if one, two, three, four, five, or more response types were coded 
per activity. 

As might be expected, foe numerical value category was coded more often than foe other response 
types. Overall and across grades, approximately 74% of foe instruction, assessment, and test preparation 
activities were coded for requiring a numerical value. For foe instruction activities, foe next most 



frequently coded categories were short answer (29%), short explanation (27%), and making a chart, 
table, or graph (24%). For the assessment activities, the next most frequently coded categories were 
short answer (3 1%), multiple choice (29%), short explanation (23%), and the requirement for students to 
show their work (22%). For die 3*^, 5^*, and 8*h grade test preparation activities short explanati on 
(65%) was die most frequently coded category, making a chart, table or graph (53%), short answer 
(38%), long explanation (34%), and show work (32%) were the next most frequently coded categories. 
As might be expected, the response types that were coded for die MSPAP test preparation activities, as 
compared to those coded for die instruction and assessment activities, are more similar to the response 
types of MSPAP tasks. It should be noted that a long explanation was defined as a paragraph or more, 
whereas a short explanation was defined as less than a paragraph and most typically, one or two 
sentences. Also, for the category ‘show work’, the activity had to include an explicit statement requiring 
the students to show their work in order to be coded as such. 
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Table Yin. 14 Response Type for Mathematics Classroom Activities - All Grades 





Number of Response Types Selected 






One 


Two 


Three 


Four 


Five 


Overall 


Instruction 


n=664 

(34%) 


n=586 

(30%) 


n=343 

(18%) 


n=187 

(10%) 


n=116 

(6%) 




Multiple Choice/Matching 


9% 


11% 


15% 


12% 


16% 


12% 


Short Answer 


7% 


29% 


40% 


52% 


64% 


29% 


Numerical Value 


60% 


75% 


84% 


90% 


91% 


74% 


Expression 


2% 


7% 


12% 


13% 


14% 


8% 


Making a Chart/Table/Graph 


9% 


16% 


29% 


55% 


63% 


24% 


Show Work 


<1% 


15% 


24% 


27% 


47% 


15% 


Short Explanation 


2% 


15% 


45% 


66% 


87% 


27% 


Long Explanation 


3 % 


6% 


9% 


23% 


37% 


10% 


Creating/Posing Math Problems 


1% 


4% 


9% 


8% 


9% 


5% 


Figural Patterns/Diagrams 


4% 


10% 


12% 


16% 


18% 


10% 


Measuring 


0% 


6% 


12% 


17% 


28% 


8% 


Modeling/Experiments 


1% 


1% 


1% 


7% 


8% 


3% 


Oral 


0% 


4% 


4% 


11% 


10% 


4% 


Other* 


3% 


2% 


4% 


4% 


9% 


45 


Assessment 


n=452 

(33%) 


n=383 

(28%) 


n=284 
( 21%) 


n=156 

(11%) 


n=71 

(5%) 


Overall 


Multiple Choice/Matching 


37% 


17% 


27% 


31% 


32% 


29% 


Short Answer 


3% 


29% 


46% 


62% 


69% 


31% 


Numerical Value 


47% 


78% 


87% 


94% 


93% 


73% 


Expression 


<1% 


9% 


14% 


17% 


31% 


11% 


Making a Chart/Table/Graph 


5% 


13% 


25% 


33% 


39% 


18% 


Show Work 


<1% 


18% 


33% 


44% 


56% 


22% 


Short Explanation 


2% 


15% 


32% 


53% 


69% 


23% 


Long Explanation 


3% 


4% 


9% 


16% 


38% 


9% 


Creating/Posing Math Problems 


<1% 


2% 


2% 


7% 


9% 


3% 


Figural Pattems/Diagrams 


<1% 


4% 


7% 


21% 


27% 


8% 


Measuring 


0% 


4% 


9% 


10% 


21% 


6% 


Modeling/Experiments 


0% 


1% 


1% 


3% 


4% 


1% 


Oral 


0% 


3% 


5% 


6% 


9% 


3% 


Other* 


2% 


3% 


3% 


5% 


3% 


3% | 



* Other: Journal, Categorizing/Ranking, and Cannot Determine 



Table Vm.14 Response Type for Mathematics Classroom Activities - All Grades (contl 





Number of Response Types Selected 


Overall 


One 


Two 


Three 


Four 


Five 


MSPAP Test Preparation 


n=15 

(12%) 


n=14 

(11%) 


n=32 

(26%) 


n=36 

(29%) 


n=15 

(12%) 


Multiple Choice/Matching 


0% 


0% 


3% 


11% 


13% 


6% 


Short Answer 


0% 


21% 


19% 


39% 


80% 


38% 


Numerical Value 


54% 


43% 


72% 


83% 


93% 


74% 


Expression 


0% 


0% 


9% 


11% 


7% 


8% 


Making a Chart/Table/Graph 


13% 


21% 


50% 


56% 


93% 


53% 


Show Work 


7% 


21% 


41% 


33% 


40% 


32% 


Short Explanation 


13% 


14% 


66% 


86% 


93% 


65% 


Long Explanation 


13% 


43% 


13% 


33% 


60% 


34% 


Creating/Posing Math Problems 


0% 


0% 


3% 


0% 


0% 


2% 


Figural Pattems/Diagrams 


0% 


7% 


13% 


28% 


7% 


18% 


Measuring 


0% 


0% 


9% 


6 % 


0% 


7% 


Modeling/Experiments 


0% 


21% 


0% 


0% 


7% 


5% 


Oral 


0% 


0% 


0% 


6% 


0% 


5% 


Other* 


0% 


7% 


3% 


9% 


7% 


9% 



* Other: Journal, Categorizing/Ranking, and Cannot Determine 



Relationship Between Classroom Activities and Questionnaire Responses for All Grades . The 
questionnaire asked teachers and students to indicate how often their classroom activities require 
explanations. Overall, 68% of die teachers indicated that they used activities requiring explanations on a 
daily (30%) or weekly (38%) basis. Whereas, 59% of the students indicated that they worked on such 
activities on a daily (23%) or weekly (26%) basis. More specifically, the questionnaire asked teachers to 
indicate how often they use multiple-choice, short answers, and written explanations on their classroom 
assessments. Approximately, 40% of die teachers indicated that they use assessment activities that 
require explanations weekly and 34% of the teachers indicated that they use assessments that require 
short answers this frequently, whereas small percentages of teachers indicated that they use multiple- 
choice formats either weekly (9%) or biweekly (14%). The results for die classroom assessment 
activities were fairly consistent with the questionnaire results, except 29% of the assessment activities 
were coded as including multiple-choice items. 

Differences Across Grades . Table Vm.15 provides die overall percentages for each grade level 
which reflect the percentage of times each of the response types was coded for an activity regardless if 
one, two, three, four, five, or more response types were coded per activity. Differences across grades 
with respect to response type tended to occur more often for the assessment and test preparation 
activities than the instruction activities. In general, for the assessment and test preparation activities, the 
numerical value category was coded more often for the middle school grades as compared to the 





elementary grades. This was most prominent for test preparation activities, in that, numerical value was 
coded for 85% of die 8 *h grade test preparation activities and 64% for the 3^ and 5*h grade test 
preparation activities. There was a slightly higher tendency for elementary assessment activities, as 
compared to middle school assessment activities, to be coded for the multiple choice category. Whereas, 
there was a slightly higher tendency for the middle school assessment activities, as compared to the 
elementary assessment activities, to be coded for die show work category. 

Several differences for die assessment activities can be observed for the on- (3^ and 5^) and off- 
( 2 nd and 4 th) elementary grades. Assessment activities from the on-grades required more short 
explanations and to some extent, more long explanations as compared to die off-grades. 

For die test preparation activities, making a table/chart/graph and creating figural patterns or 
diagrams as the final product occurred more often for 3 rc * grade as compared to die 5^ and 8 ^ grades. 
Whereas, long explanations occurred more often for die 8 *h grade test preparation activities (41%) as 
compared to the 3 r< * (23%) and 5^ grades (32%). 





All 


Grade 


Instruction 


grades 


2nd 


3ItI 


4to 


5® 


7tll 


8th 


Multiple Choice/Matching 


12% 


13% 


12% 


15% 


14% 


7% 


9% 


Short Answer 


29% 


29% 


28% 


33% 


31% 


29% 


25% 


Numerical Value 


74% 


67% 


66% 


78% 


77% 


82% 


80% 


Expression 


8% 


7% 


10% 


5% 


5% 


8% 


11% 


Making a Chart/Table/Graph 


24% 


23% 


26% 


24% 


30% 


17% 


21% 


Show Work 


15% 


10% 


18% 


16% 


13% 


14% 


20% 


Short Explanation 


27% 


18% 


28% 


27% 


31% 


28% 


29% 


Long Explanation 


10% 


7% 


9% 


7% 


10% 


11% 


14% 


Creating/Posing Math Problems 


5% 


4% 


7% 


6% 


6% 


5% 


2% 


Figural Patterns/Diagrams 


10% 


11% 


11% 


10% 


10% 


6% 


10% 


Measuring 


8% 


7% 


8% 


5% 


8% 


10% 


10% 


Modeling/Experiments 


3% 


2% 


2%. 


4% 


3% 


4% 


4% 


Oral 


4% 


6% 


7% 


5% 


1% 


3% 


1% 


Other* 


4% 


8% 


3% 


4% 


4% 


3% 


2% 


Assessment 


Multiple Choice/Matching 


29% 


33% 


27% 


38% 


35% 


22% 


20% 


Short Answer 


31% 


27% 


27% 


40% 


32% 


38% 


29% 


Numerical Value 


73% 


65% 


73% 


70% 


65% 


85% 


77% 


Expression 


11% 


7% 


10% 


5% 


7% 


19% 


17% 


Making a Chart/Table/Graph 


18% 


20% 


19% 


12% 


22% 


12% 


21% 


Show Work 


22% 


14% 


21% 


19% 


15% 


28% 


33% 


Short Explanation 


23% 


16% 


37% 


16% 


25% 


20% 


21% 


Long Explanation 


9% 


5% 


7% 


3% 


8% 


10% 


18% 


Creating/Posing Math Problems 


3% 


3% 


3% 


3% 


3% 


3% 


1% 


Figural Pattems/Diagrams 


8% 


7% 


13% 


6% 


11% 


5% 


6% 


Measuring 


6% 


8% 


8% 


6% 


8% 


5% 


2% 


Modeling/Experiments 


1% 


1% 


1% 


1% 


2% 


1% 


1% 


Oral 


3% 


6% 


4% 


2% 


2% 


2% 


2% 


Other* 


3% 


2% 


7% 


2% 


3% 


2% 


1% 


MSPAP Test Preparation 


Multiple Choice/Matching 


6% 


— 


8% 


— 


11% 


— 


2% 


Short Answer 


38% 


— 


46% 


— 


29% 


— 


38% 


Numerical Value 


74% 


— 


64% 


— 


64% 


— 


85% 


Expression 


8% 


— 


3% 


— 


14% 


— 


9% 


Making a Chart/Table/Graph 


53% 


— 


64% 


— 


46% 




48% 


Show Work 


32% 


— 


39% 


— 


29% 




29% 


Short Explanation 


65% 


— 


69% 


— 


71% 




59% 


Long Explanation 


34% 


— 


23% 


— 


32% 


— 


41% 


Creating/Posing Math Problems 


2% 


— 


0% 


— 


4% 


— 


3% 


Figural Pattems/Diagrams 


18% 


— 


31% 


— 


7% 


— 


14% 


Measuring 


7% 


— 


5% 


— 


4% 


— 


10% 


Modeling/Experiments 


5% 


— 


0% 


— 


8% 


— 


8% 


Oral 


5% 


— 


8% 


— 


.0% 


— 


5% 


Other* 


9% 


— 


13% 


— 


15% 


— 


3% 




127 



131 



Relationship Between Classroom Activities and Questionnaire Responses Across Grades . These 
differences for die grade levels are somewhat consistent with the results from die questionnaire. 
Teachers and students were asked to indicate the extent to which their classroom activities required 
explanations and teachers were also asked to indicate die extent to which their assessments required 
written explanations. A larger percentage (52%) of 3^ and 5*h grade teachers reported that on a weekly 
basis they use assessment activities that require written explanations, as compared to 2 Q d and grade 
teachers (40%), 7^ grade teachers (18%), and 8^ grade teachers (27%). Also, larger percentages of 4*h 
(28%) and 5* (32%) grade students indicated they work on assessments that ask them to explain in 
writing how they got their answer on a weekly basis, while only 1 1% of the 7^ and 16% of the 8*h grade 
students indicated this level of frequency. Further, a larger percentage of 3^ and 5*h grade teachers 
reported that on a daily or weekly basis they use classroom activities that require students to explain 
(82%) as compared to 2°d and 4^ (68%), 7*h (58%) and 8^ (50%) grade teachers. 

Similarity to MSPAP Tasks 

Each of the classroom instruction, assessment, and MSPAP test preparation activities were coded 
with respect to their similarity to MSPAP tasks. In particular, the process needed for solution, the type 
of responses required of students, and the format of the responses were considered in order to classify the 
activities according to one or more MSPAP-like levels. The first two levels include those activities that 
were considered “not at all like MSPAP”: 1) computations, estimations, and equations, and 2) traditional 
textbook-like word problems. The first category reflects those problems that solely ask students to do a 
computation or estimation, or to solve an equation. The problems in die second category reflect 
traditional word problems in which students need to provide or select a numerical answer based on their 
computations. Thus, the first two categories do not require the same level of problem solving and/or 
reasoning as defined by the MLO’s and MSPAP. Although some of the skills required in problems of 
these types may also be required by MSPAP tasks, overall the problems themselves are not considered to 
be similar to MSPAP tasks. 

The other four levels include activities that are similar to MSPAP tasks to some extent: MSPAP-like 
1, MSPAP-like 2, MSPAP-like 3, and MSPAP-like 4. Activities at the MSPAP-like 1 level only require 
students to develop or complete a graph, table, pattern, or to physically measure an object. In these types 
of activities, students are not required to provide any interpretation or explanation of their work, and tire 
activity does not require the same level of problem solving and/or reasoning as defined by the MLO’s 
and required by the MSPAP tasks. 



MSPAP-like 2 activities require some problem solving and/or reasoning, but not to the same extent 
as required by MSPAP tasks. They also require students to show their work, provide an explanation, 
and/or interpret tables or graphs, and they can be completed in about five minutes. 

MSPAP-like 3 and 4 activities require a similar level of problem solving and/or reasoning as 
required by the MSPAP tasks. MSPAP-like 3 tasks also require at least two short explanations or one 
long explanation (i.e., about a paragraph), and consist of about 3-5 items related to the same problem 
situation. Many of them also asked students to develop graphs, tables, or charts. MSPAP-like 3 tasks 
are considered to be similar to MSPAP tasks in terms of the processes being measured and the format, 
but not as extensive in length. MSPAP-like 4 tasks are considered to be similar to MSPAP tasks in 
terms of the processes being measured as well as the format in which they are measured. These tasks 
require students to show their work and/or to develop graphs, tables, or charts; they require at least 3 
short explanations and/or one or more long explanations; and require students to respond to 6 or more 
items related to the same situation. 

Each activity, regardless if it was one task or a set of distinct items, could be coded in more than one 
of die six MSPAP-like levels. For example, of the 1,940 instruction activities, 83% (1,617) were coded 
solely for one MSPAP-like level, 14% were coded for two MSPAP-like levels, and 2% were coded for 
three MSPAP-like levels. The other 1% were coded for four or more MSPAP-like levels. The test 
preparation activities were similar in this regard, about 89% were coded solely in one MSPAP-like level. 
A smaller percentage of assessment activities were coded in only one level (62%), and approximately 
27% were coded in two levels, 9% in three levels, and 2% in four levels. It should be noted that for 
activities coded in two or more MSPAP-like levels, approximately 90% of the activities included at least 
one level that was “not at all like MSPAP”. 

All Grades . Table VIQ.16 indicates the percentage of times an activity was coded for each MSPAP- 
like level when one, two, and three levels were coded per artifact. The last column in the table, labeled 
‘overall’, indicates the percentage of times an activity was coded for each level regardless if one, two, or 
three levels were coded per activity. For example, for those instruction activities in which only one level 
was selected, computation/equation was selected for 39% of them; for those activities in which two 
levels were selected, computation/equation was selected for 79% of the tasks; and for those activities in 
which three levels were selected, computation/equation was selected for 98% of the tasks. Overall, 
regardless of how many MSPAP-like levels were coded for an activity, computation/equation was 
selected for 46% of the instruction activities. 

As indicated in the table the most common type of instruction and assessment activity required the 
student to perform computations, to estimate, or to solve equations. This level was selected for 46% of 
the instruction activities and 66% of the assessment activities. In general, the MSPAP-like 2 was the 
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next most commonly coded level for the instruction (34%) and assessment (32%) activities and then, 
traditional word problems (14% for instruction and 31% for assessment activities). The MSPAP-like 4 
level was one of the least frequently coded categories for instruction (5%) and assessment (4%) 
activities. 

As might be expected, the 3 r ^, 5^, and 8& grade MSPAP test preparation activities, as compared to 
the instruction and assessment activities, are more similar to MSPAP tasks as indicated by the 'overall’ 
column. The most frequently coded levels for the test preparation activities were MSPAP-like 2 (38%) 
and MSPAP-like 4 (37%) task types. The next most frequently coded task type for test preparation 
activities was the MSPAP-like 3 task type (27%). The computation/equation level was selected for only 
15% of the MSPAP test preparation activities. Similar tables for each of die grades are provided in 
Appendix E. 



Table VHI.16 MSP AP-like Levels for Mathematics Classroom Activities - All Grades 





Number of Levels Selected 


Overall 


One 


Two 


Three 


Instruction 


n=1617 (83%) 


n=278 (14%) 


n=41 (2%) 


Not at all like MSPAP 










Computation/Equation 


39% 


79% 


98% 


46% 


Traditional Word Problems 


6% 


49% 


73% 


14% 


MSPAP-like Levels 










MSPAP-like 1 


10% 


19% 


37% 


12% 


MSPAP-like 2 


31% 


45% 


88% 


34% 


MSPAP-like 3 


| 9% 


7% 


2% 


9% 


MSPAP-like 4 


5 % 


<1% 


2% 


5% 


Assessment 


n=857(62%) 


n=388(27%) 


n=129(9%) 


overall 


Not at all like MSPAP 










Computation/Equation 


49% 


91% 


98% 


66% 


Traditional Word Problems 


4% 


67% 


92% 


31% 


MSPAP-like Levels 










MSPAP-like 1 


6% 


13% 


24% 


12% 


MSPAP-like 2 


27% 


26% 


83% 


33% 


MSPAP-like 3 


9% 


3% 


2% 


6% 


MSPAP-like 4 


6% 


0% 


0% 


4% 


MSPAP Test Preparation 


n=115 (92%) 


n=7 (6%) 


n=2 (2%) 


overall 


Not at all like MSPAP 










Computation/Equation 


9% 


100% 


50% 


15% 


Traditional Word Problems 


0% 


14% 


50% 


2% 


MSPAP-like Levels 










MSPAP-like 1 


3% 


0% 


50% 


4% 


MSPAP-like 2 


33% 


86% 


100% 


38% 


MSPAP-like 3 


23% 


0% 


50% 


27% 


MSPAP-like 4 


32% 


0% 


0% 


37% 



Relationship Between Classroom Activities and Questionnaire Responses for All Grades . On 
the questionnaires, teachers and 4 th, 5 ^ 7 th, and 8th students were asked to indicated how often 
students have the opportunity to solve tasks similar to those on MSPAP. Overall, 31% of the teachers 
indicated that students have the opportunity weekly or daily, 47% indicated biweekly or monthly, 21% 
indicated about 4 times per year, and 2% said never. Overall, 38% of the students indicated that they 
solve such tasks weekly or daily, 33% indicated biweekly or monthly, 22% indicated about 4 times per 
year, and 7% said never. The analysis of the classroom instruction materials indicated that for those 
activities in which only one MSPAP-like level was coded, 55% of them were coded as being on the 
MSPAP scale ranging from MSPAP-like 1 to MSPAP-like 4. On the other hand, only 14% received the 
two highest levels (MSPAP-like 3 and MSPAP-like 4). These results indicate that over 50% of the 
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instruction activities required problem solving and/or reasoning as defined by the MLO’s to some extent 
A smaller percentage of die activities, however, were as extensive as the MSPAP tasks and required die 
same level of problem solving. 

Teachers and students were asked to indicate on their questionnaires die extent to which they use 
open-ended problems and extended activities for both instruction and assessment The majority of die 
classroom instruction and assessment activities that were coded MSPAP-like 1, MSPAP-tike 2, and 
MSPAP- like 3 could be classified as open-ended problems, whereas activities that were coded as 
MSPAP-like 4 could be classified as open-ended problems that require an extended period of time to 
complete. Approximately 50% of the instruction activities and 42% of the assessment activities received 
codes ranging from MSPAP-like 1 to MSPAP-like 3, and 5% of the instruction activities and 6% of the 
assessment activities were coded as MSPAP-like 4. Overall, 59% of the teachers indicated that students 
have the opportunity daily or weekly to work on open-ended problems for instruction and 25% indicated 
this level of frequency for assessment In addition, 18% of the teachers indicated that students have the 
opportunity daily or weekly to work on extended problems for instruction and 8% indicated this level of 
frequency for assessment. In general, the results from the classroom activities are fairly consistent with 
the teacher questionnaire results, in particular, with respect to the assessment activities. 

The percentages of students who indicated that they had the opportunity to work on open-ended 
problems tended to be higher than the teacher percentages. Overall, 77% of the students indicated that 
they had the opportunity to work on open-ended problems for instruction on a daily or weekly basis and 
57% indicated this level of frequency for assessment Also, 1 8% of the students indicated that they had 
the opportunity to work on extended problems for instruction on a daily or weekly basis and 3% 
indicated this level of frequency for assessment 

Differences Across Grades. In Table Vm.17, the overall results for each grade level are provided. 
The overall percentages in the table reflect the percentage of times each MSPAP-like level was coded for 
an activity regardless of the number of codes per activity. Differences across grades were rather small 
For instruction, slightly more elementary activities were coded as MSPAP-like 1 and slightly more 
middle school activities were coded as MSPAP-like 3 activities. Also, for instruction and a s s ess m e n t 
activities, there was a slight increase in the percentage of MSPAP-like 3 and MSPAP-like 4 activities for 
the on-grade levels when compared to the off-grade levels. As an example, the percentage of MSPAP- 
like 3 and 4 instruction activities for the on-grades (3 rd , 5 th , and 8 th ) range from 14% to 20% depending 
on grade, whereas the percentages for the off-grades range from 8% to 13%. 

With regard to MSPAP test preparation, 58% of the 5^* grade activities were coded as MSPAP-like 
2, whereas only 33% of the 3 rd and 8^ grade activities were coded at this level. More activities in the 
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3 rc * and 8 * grades (57% and 53%) were coded as either MSPAP-like 3 or MSPAP-Iike 4 compared to a 
smaller percentage of activities in die 5th grade (40%). 

Relations hip Between Classroom Activities and Questionnaire Responses Across Grades . The 
results from die questionnaires are somewhat consistent in that 3 r ^ and 5 th grade teachers inHjr-ated that 
they use tasks similar to those on MSPAP most frequently and 7 th grade teachers indicated that they use 
them die least frequendy. Also, a smaller percentage of 4* and 7 th grade students (35% and 31%, 
respectively) reported that they worked on MSPAP-like tasks daily or weekly when compared to the 5 th 
and 8 th grade students (44% and 40%, respectively). 



Table VIII.17 MSPAP-like Levels for Mathematics Classroom Activities — For Each Grade 



Instruction 


All 


Grade 


grades 


2 nd 


3 rd 


4 tn 


5™ 


ytn 


8 th 


Not at all like MSPAP 
















Computation/Equation 


46% 


51% 


44% 


50% 


41% 


45% 


46% 


Traditional Word Problems 


14% 


6 % 


12 % 


17% 


17% 


15% 


17% 


MSPAP-like Levels 
















MSPAP-like 1 


12 % 


17% 


12 % 


11 % 


15% 


6 % 


9% 


MSPAP-like 2 


34% 


28% 


35% 


38% 


38% 


36% 


30% 


MSPAP-like 3 


9% 


5% 


8 % 


8 % 


8 % 


11 % 


13% 


MSPAP-like 4 


s* . 


h 3% 


6 % 


1 % 


7% 


2 % 


7% 


Assessment 


Not at all like MSPAP 
















Computation/Equation 


66 % 


64% 


54% 


72% 


62% . 


79% 


67% 


Traditional Word Problems 


31% 


34% 


26% 


38% 


35% 


34% 


23% 


MSPAP-like Levels 
















MSPAP-like 1 


12 % 


16% 


10 % 


12 % 


12 % 


j_9% 


11 % 


MSPAP-like 2 


32% 


31% 


40% 


31% 


35% 


34% 


24% 


MSPAP-like 3 


6 % 


4% 


8 % 


3% 


6 % 


5% 


11 % 


MSPAP-like 4 


4% 


1 % 


5% 


1 % 


7% 


2 % 


4% 


MSPAP Test Preparation 


Not at all like MSPAP 
















Computation/Equation 


15% 


— 


10 % 


— 


11 % 


— 


21 % 


Traditional Word Problems 


2 % 


— 


3% 


— 


4% 


— 


2 % 


MSPAP-like Levels 
















MSPAP-like 1 


4% 


— 


5% 


— 


4% 


— 


3% 


MSPAP-like 2 


| 38% 


— 


33% 


— 


58% 


— 


33% 


MSPAP-like 3 


27% 


— 


26% 


— 


11 % 


— 


24% 


MSPAP-like 4 


37% 


— 


31% 


— • 


29% 


— 


29% 



Nature of Activi ties Within Each MSPAP-like Level. To provide more information about the 
nature of the activities in each of the six MSPAP-like levels, the instruction and assessment activities that 
were coded in only one level were analyzed with respect to format, process outcomes, content outcomes. 



133 

137 



interpretation of graphs, integration, manipulates, and response type. Of the 1,940 instruction 
activities, 1,617 received only one code for MSPAP-like level; and of the 1,388 assessment activities, 
857 received only one code. The results are reported for the grades combined. 

Activity Format Nearly all of die computation/equation and traditional word problem activities 
were coded as containing several distinct items. Whereas, all of the MSPAP-like 3 and MSPAP-like 4 
activities and ova- 90% of die MSPAP-like 2 activities were coded as being one task with several items 
related to die same problem situation. 

Maryland Learning Process Outcomes . As defined by the coding scheme, none of the 
computation/ equation or traditional word problem instruction and assessment activity types were coded 
for problem solving and/or reasoning, about half of the MSPAP-like 1 activities were coded for problem 
solving and/or reasoning, and all MSPAP-like 2 through 4 activities were coded for problem solving 
and/or reasoning. It is important to note that the extent to which problem solving and/or reasoning was 
reflected in the activities increased as the level of die MSPAP-like activities increases. 

Maryland Learning Content Outcomes . As indicated in Table VIII. 18, the most frequently coded 
content outcome for the computation/equation, traditional word problem, MSPAP-like 2, and MSPAP- 
like 3 instruction and assessment activity types was numbers and operations. However, for the MSPAP- 
like 2 and MSPAP-like 3 activities, as compared to the other two more traditional activity levels, other 
content activities were coded more frequently, such as geometry/measurement and statistics. Moreover, 
statistics was coded more frequently than the other content areas for MSPAP-like 4 activities, with 
numbers and operations and geometry/measurement being the next most frequently coded content 



outcomes. 



Table VIII. 1 8 Content Learning Content Outcomes for MSPAP-like Levels 





Not at all like MSPAP 


MSPAP-like Levels 




Computation 
or Equation 


Traditional 

Word 

Problems 


MSPAP 
like 1 


MSPAP 
-like 2 


MSPAP 
-like 3 


MSPAP 
-like 4 


Instruction 


n=627 


n=99 


n=154 


n=494 


n=149 


n=85 


Numbers and Operations 


70% 


90% 


12% 


55% 


41% 


47% 


Geometry/ Measurement 


21% 


16% 


44% 


20% 


34% 


44% 


Statistics 


2% 


5% 


38% 


27% 


39% 


49% 


Probability 


3% 


5% 


2% 


12% 


14% 


19% 


Patterns and Relationships 


1% 


0% 


15% 


4% 


6% 


4% 


Prealgebra 


4% 


0% 


0% 


<1% 


1% 


0% 


Functions and Algebra 


8% 


18% 


0% 


3% 


2% 


2% 


None 


0% 


0% 


0% 


0% 


0% 


0% 


Assessment 


n=416 


n=33 


n=55 


n=215 


n=74 


n=50 


Numbers and Operations 


71% 


85% 


11% 


58% 


49% 


36% 


Geometry/Measurement 


22% 


12% 


40% 


21% 


30% 


40% 


Statistics 


2% 


0% 


38% 


28% 


38% 


60% 


Probability 


2% 


0% 


2% 


8% 


10% 


14% 


Patterns and Relationships 


1% 


6% 


13% 


5% 


4% 


4% 


Prealgebra 


9% 


3% 


0% 


1% 


0% 


0% 


Functions and Algebra 


16% 


15% 


2% 


3% 


4% 


8% 


None 


0% 


0% 


4% 


0% 


0% 


0% 



Interpretation of Graphs , Tables or Charts . The interpretation of graphs, tables, or charts was rare for 
instruction and assessment activities coded as either computation/equation or traditional word problems 
(between 4% and 12%). Whereas, the interpretation of graphs, tables dr charts became more prevalent as 
the level of the MSPAP-like activities increased as indicated in Table Vin.19. As an example, consider 
the instruction activities. Approximately 44% of the MSPAP-like 2 activities, 69% of the MSPAP-like 3 
activities, and 85% of the MSPAP like 4 activities required interpretation of graphs, tables or charts. 



Table VDI.19 Interpreting Graphs. Tables, or Charts for MSPAP-like Levels 





Not at all like MSPAP 


MSPAP-like Levels 




Computation 
or Equation 


Traditional 

Word 

Problems 


MSPAP 
-like 1 


MSPAP 
-like 2 


MSPAP 
-like 3 


MSPAP 
- like 4 


Instruction 


n=627 


n=99 


n=154 


n=494 


n=149 


n=85 


Yes 


j 5% 


7% 


8% 


44% 


69% 


85% 


No 


95% 


93% 


92% 


56% 


31% 


15% 


Assessment 


n=416 


n=33 


n=55 


n=215 


n=74 


n=50 


Yes 


4% 


12% 


16% 


47% 


66% 


78% 


No 


96% 


88% 


84% 


54% 


34% 


22% 
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Integration with Other Subject Areas . In general, as the level of the MSPAP-like activities increases, 
the percentage of the tasks that were integrated with another subject area also increases as indicated in 
Table Vm.20. For example, integration was prevalent for only 1% of die computation/equation 
instruction activities, 16% of the MSPAP-like 2 instruction activities, and 60% for the MSPAP-like 3 
instruction activities. 

Table Vin.20 Integration of Other Subject Areas for MSPAP-like Levels 





NotataUlikeMSPAP 


MSPAP-like Levels 




Computation 
or Equation 


Traditional 

Word 

Problems 


MSPAP 
-like 1 


MSPAP 
-like 2 


MSPAP 
-like 3 


MSPAP 
- like 4 


Instruction 


n=627 


n=99 


n=154 


n=494 


n=149 


h=85 


Science 


<1% 


2% 


1% 


3% 


9% 


21% 


Social Studies 


1% 


1% 


2% 


1% 


2% 


3% 


Reading Materials 


1% 


0% 


1% 


3% 


1% 


2% 


Writing Letters/Persuasive 
Paragraphs, etc. 


0% 


0% 


0% 


5% 


14% 


33% 


Language Usage 


0% 


0% 


0% 


1% 


1% 


0% 


Art 


0% 


0% 


1% 


1% 


3% 


2% 


Glyphs/Tesselations 


0% 


0% 


12% 


3% 


1% 


0% 


No Integration 


99% 


97% 


83% 


84% 


69% 


40% 


Assessment 


n=416 


n=33 


a=55 


n=215 


n=74 


n=50 


Science 


<1% 


0% 


4% 


3% 


4% 


13% 


Social Studies 


0% 


0% 


2% 


2% 


3% 


0% 


Reading Materials 


1% 


0% 


0% 


1% 


1% 


0% 


Writing Letters/Persuasive 
Paragraphs, etc. 


0% 


0% 


0% 


6% 


20% 


40% 


Language Usage 


0% 


0% 


0% 


1% 


0% 


0% 


Art 


<1% 


0% 


2% 


2% 


0% 


0% 


Glyphs/Tesselations 


0% 


0% 


8% 


2% 


3 % 


0% 


No Integration 


99% 


100% 


85% 


84% 


68% 


48% 



Use of Maninulatives. Calculators, and Computers . In general, as the level of the MSPAP-like 
activities increases, the use of manipulates and calculators increases as indicated in Table VIII. 21. For 
example, the use of manipulates was prevalent for 16% of the computation/equation instruction 
activities, 31% of the MSPAP-like 2 instruction activities, and 49% for the MSPAP-like 4 instruction 
activities. 
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Table VHI.21 Manipulatives. Calculators, and Computers for MSPAP-like Levels 





Not at all like MSPAP 


MSPAP-like Levels 




Computation 
or Equation 


Traditional 

Word 

Problems 


MSPAP 
-like 1 


MSPAP 
-like 2 


MSPAP 
-like 3 


MSPAP 
-like 4 


Instruction 


n=627 


n=99 


n=154 


n=494 


n=149 


n=85 


Manipulatives 


3% 


1% 


2% 


31% 


42% 


49% 


Calculators 


8% 


7% 


12% 


2% 


4% 


15% 


Computers 


2% 


1% 


3% 


<1% 


1% 


| 1% 


None 


87% 


91% 


81% 


66% 


53% 


34% 


Assessment 


n=416 


n=33 


n=55 


n=215 


n=74 


n=50 


Manipulatives 


3% 


0% 


6% 


19% 


23% 


30% 


Calculators 


4% 


6% 


15% 


3% 


10% 


7% 


Computers 


1% 


0% 


4% 


1% 


0% 


2% 


None 


92% 


94% 


76% 


77% 


68% 


61% 



Response Type Required of Student . In general, the most frequently coded response types across die 
MSPAP-like levels differed for activities that were not similar to MSPAP tasks and for those that were 
similar as shown in Table VIQ.22. For example, the most frequently coded response types for 
computation/equation instruction activities were numerical value (78%), short answer (24%), multiple- 
choice (15%), and show work (10%). The most frequently coded response types for MSPAP-like 2 and 
MSPAP-like 3 activities were numerical value (67% and 71%), short explanation (43% and 73%), 
making a chart, graph, or table (31% and 56%), and short answer (33% and 42%), with the latter three 
response types being more prevalent for MSPAP-like 3 activities. The most frequently coded response 
types for MSPAP-4 activities were short explanation (92%), numerical value (81%), making a chart, 
graph, or table (72%), and long explanations (61%). It is important to note that as the level of MSPAP- 
like activities increases, more response types were coded for a given activity and the response types were 
more similar to the response types on MSPAP tasks. 
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Table Vm.22 Response Type for MSPAP-like Levels 





Not at all like MSPAP 


MSPAP-like Levels 




Computation 
or Equation 


Traditional 

Word 

Problems 


MSPAP 
-like 1 


MSPAP 
-like 2 


MSPAP 
-like 3 


MSPAP 
- like 4 


Instruction 


n=627 


n=99 


n=154 


n=494 


n=149 


n=85 


Multiple Choice/Matching 


15% 


16% 


3% 


6% 


4% 


5% 


Short Answer 


24% 


8% 


14% 


33°/* 


42% 


42% 


Numerical Value 


78% 


85% 


42% 


67% 


71% 


81% 


Expressions 


7% 


7% 


3% 


6% 


9% 


13% 


Making a Chart/Table/Graph 


2% 


1% 


49% 


31% 


56% 


72% 


Show Work 


10% 


25% 


3% 


16% 


25% 


35% 


Short Explanation 


2% 


1% 


1% 


43% 


73% 


92% 


Long Explanation 


0% 


1% 


1% 1 


11% 


38% 


61% 


Creating Math Problems 


1% 


3% 


3% 


1% 


3% 


4% 


Figural Patterns/Diagrams 


9% 


0% 


21% 


7% 


12% 


9% 


Measuring 


1% 


0% 


25% 


5% 


13% 


25% 


Modeling/Expressions 


0% 


0% 


8% 


2% 


9% 


19% 


Oral 


4% 


0% 


1% 


6% 


5% 


7% 


Other 


3% 


1% 


1% 


5% 


5% 


8% 


Assessment 


n=416 


n=33 


n=55 


n=215 


n=74 


n=50 


Multiple Choice/Matching 


23% 


15% 


6% 


8% 


5% 


8% 


Short Answer 


26% 


30% 


15% 


25% 


42% 


56% 


Numerical Value 


82% 


85% 


40% 


61% 


74% 


90% 


Expressions 


11% 


6% 


0% 


7% 


8% 


30% 


Making a Chart/Table/Graph 


2%. 


9% 


55% 


33% 


45% 


80% 


Show Work 


17% 


15% 


9% 


24% 


38% 


38% 


Short Explanation 


2% 


3% 


0% 


49% 


66% 


96% 


Long Explanation 


0% 


0% 


0% 


14% 


55% 


62% 


Creating Math Problems 


<1% 


0% 


2% 


7% 


3% 


4% 


Figural Pattems/Diagrams 


5% 


0% 


6% 


9% 


23% 


16% 


Measuring 


1% 


0% 


26% 


3% 


11% 


6% 


Modeling/Expressions 


1% 


0% 


7% 


2% 


3% 


8% 


Oral 


5% 


3% 


2% 


3% 


7% 


6% 


Other 


1% 


3% 


4% 


8% 


4% 


12% 



Results from Classroom Scoring Schemes 



The scoring schemes were coded according to the type of scoring scheme (e.g., letter/numerical 
grades, checklist, point system, scale levels); whether the scoring scheme was specific to an activity or a 
more general scheme that applied to a variety of activities; the components of the response that were 
being evaluated, and the evaluator of student performance. As mentioned previously, a total of 332 
scoring schemes were coded with the majority of them being teacher developed (35%), county developed 
(20%), or commercially developed (13%). 

Scoring Scheme Type 

The scoring schemes were coded as to whether they reflected the use of letter/numerical grades (e.g., 
90-100%, 89-90%, etc.), checklists, point systems, scale levels, or a qualitative written explanation. If a 
scoring scheme was a point system, it was coded as to whether dimensions were specified. A point 
system without dimensions specified a certain number of points for correct answers to the activity, 
whereas a point system with dimensions indicated the number of points to be assigned to each part of the 
response, for example two points for die numerical answer and three points for the explanation. Scoring 
schemes that were coded as having scale levels were used to evaluate the response to an activity 
holistically, that is, one overall score was assigned to the entire response. When a scoring scheme was 
classified as consisting of scale levels, it was coded according to the number of scale levels (only 2 levels 
or more than 2 levels) and to die type of criteria for each level (no/low-level criteria or high-level 
criteria). Low-level criteria were defined as focusing primarily on the inclusion or omission of the 
answer, explanation, or graph without evaluating the quality of the response. High-level criteria were 
defined as focusing primarily on the quality of die response (i.e., explanation or graph). Qualitative 
written explanations reflect scoring schemes that provided only a written evaluation of the response 
without providing any quantitative score. 

Table Vm.23 shows that over half of the scoring schemes across all grades (38%) were coded as 
consisting of more than 2 scale levels with either high or low criteria, 13% were point systems with 
dimensions, 9% were letter/numerical grades, 8% were point systems without dimensions, and 7% were 
checklists. Within some of the scoring scheme types, there were differences across grades. For instance, 
a larger percentage of scoring schemes in the 2 n< ^ and 4*h grades (off-grades) compared to the other 
grade levels were coded as letter/numerical grades. Whereas, a larger percentage of scoring schemes in 
the 3 f d and 5^ grades (on-grades) compared to the other grades were coded as more than 2 scale levels 
with high-level criteria. Lastly, the 7*h and 8& grades contained a higher percentage of point systems 
with dimensions than the elementary grades. 
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Table VI1I.23 Scoring Schemes Used in Mathematics Classroom Activities 





All Grades 


2 nd 


3 rd 


“pi - 


"5®“ 


7 th 


8 th 


Letter/ Numerical grades 


9% 


32% 


4% 


16% 


7% 


5% 


4% 


Checklist 


7% 


6 % 


8 % 


19% 


8 % 


6 % 


0 % 


Point System - No Dimensions 


8 % 


0 % 


4% 


13% 


8 % 


6 % 


17% 


Point System - With Dimensions 


15% 


6 % 


4% 


0 % 


10 % 


28% 


30% 


Only 2 Scale Levels 


1 % 


3% 


1 % 


0 % 


2 % 


0 % 


0 % 


More than 2 Scale Levels 
(no or low-level criteria) 


8 % 


6 % 


11 % 


0 % 


7% 


10 % 


9% 


More than 2 Scale Levels 
(high-level criteria) 


50% 


44% 


68 % 


44% 


57% 


44% 


36% 


Qualitative Written Explanation 


2 % 


0 % 


0 % 


3% 


2 % 


2 % 


3% 


Cannot Determine 


1 % 


3% 


0 % 


6 % 


0 % 


0 % 


2 % 



General versus Specific Scoring Schemes 

Scoring schemes were classified according to whether they pertained to only one specific activity or 
if they were applicable to more than one activity. Across all grades. Table Vm.24 indicates that 61% of 
the scoring schemes were specific to an activity, and the other 39% were general. Differences occurred 
in the percentages for 4 th and 7 th grade compared to the other grade levels. For the 2 n d, 3^, 5^, and 
8 th grades the majority of the scoring schemes were for a specific task, whereas in 4& grade the reverse 
was true, and in 7th grade there was an equal percentage of specific and general scoring schemes. 



Table VTTT.24 General versus Specific Scoring Schemes 





All Grades 


2 nd 


3 rd 


4 th 


“5®“ 


7 th 


8 th 


Scoring scheme for a 
specific activity 


61% 


71% 


67% 


34% 


69% 


50% 


67% 


General scoring scheme 


39% 


30% 


31% 


66 % 


32% 


50% 


33% 



Evaluation Components 

All Grades . The scoring schemes that specified either dimensions or scale levels with criteria were 
analyzed according to the type of components that were reflected in the evaluation criteria. In addition to 
mathematics knowledge. Table Vm.25 indicates other evaluation components that were coded for those 
scoring schemes that explicitly specified evaluation criteria. One or more evaluation components could 
have been coded for a given scoring scheme. Besides mathematics knowledge which was embedded in 
these scoring schemes, the most frequently coded evaluation component was communication in writing 



and/or explanations (63%). The next most frequently coded evaluation components were solution 
strategies/work (34%), graphs, tables, or charts (33%), and language usage (26%). 



Table Y1II.2S Evaluation Components in Scoring Schemes - All Grades 





Number of Evaluation Components Selected 


Overal 

1 


One or 
None 


Two 


Three 


Four 


Five 


n=61 

(18%) 


n=67 

(20%) 


n=l 10 
(33%) 


n=70 

(21%) 


n=22 

(7%) 


Communication in 
Writing/Explanation 


0% 


49% 


81% 


93% 


100% 


63% 


Language Usage 


0% 


6% 


24% 


53% 


77% 


26% 


Solution Strategies/Show Work 


0% 


30% 


40% 


51% 


59% 


34% 


Graphs/Tables/Charts, etc. 


3% 


24% 


31% 


59% 


73% 


33% 


Examples 


0% 


0% 


8% 


17% 


18% 


8% 


Oral 


0% 


2% 


2% 


1% 


14% 


3% 


Appearance/Neatness 


0% 


3% 


16% 


21% 


50% 


14% 


Group Skills 


2% 


2% 


2% 


6% 


9% 


3% 


Attitudes 


3% 


0% 


0% 


0% 


0% 


1% 



Differences Across Grades. Scoring schemes in the middle grades as well as die elementary on- 
grades (3”* and 5^), as compared to the elementary off-grades (2 n< * and 4^), were more likely to 
include written explanations as an evaluation component as shown in Table VIII.26. For e xam ple, 
consider the two extremes: 73% of the 3*^ grade scoring schemes were coded for communication in 
writing/explanations, whereas only 44% of the 4 th grade scoring schemes were so coded. Another 
difference across grades was the evaluation of graphs, tables, and charts. As the grade level increased, 
the percentage of scoring schemes that included this evaluation component increased. For example, 1 8% 
in 2 n d grade compared to 44% in 8^ grade. 
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Table vm.26 Evaluation Components in Scoring Schemes -- For Each Grade 





All 


Grade 




grades 


2 nd 


3 rt 


4 th 


5 ®- 


7 tn 


g® 


Communication in Writing/Explanation 


63% 


47% 


73% 


44% 


62% 


70% 


64% 


Language Usage 


26% 


15% 


31% 


16% 


31% 


36% 


17% 


Solution Strategies/Show Work 


34% | 


27% 


29% 


47% 


28% 


41% 


38% 


Graphs/Tables/Charts, etc. 


33% 


18% 


24% 


25% 


36% 


42% 


44% 


Examples 


8 % 


9% 


0 % 


3% 


10 % 


11 % 


14% 


Oral 


3% 


3% 


4% 


0 % 


2 % 


3% 


3% 


Appearance/Neatness 


14% 


6 % 


9% 


19% 


12 % 


22 % 


17% 


Group Skills 


3% 


3% 


1 % 


9% 


0 % 


6 % 


3% 


Attitudes 


1 % 


0 % 


1 % 


3% 


0 % 


0 % 


0 % 


Not Applicable 


11 % 


27% 


4% 


22 % 


18% 


8 % 


5% 



Evaluator of Student Performance 

The scorin g schemes were also coded according to the person(s) applying the scheme to the student 
responses, either die teacher, the student, peers, or some combination. As shown in Table VHI.27, across 
grade levels the teachers most often were the evaluators (91%), then the student (4%), a combination of 
teacher and student and/or peers (3%), and peers (2%). The results across grades were similar. 



Table VIII .27 Scoring Evaluator of Mathematics Classroom Activities 





All Grades 


2 nd 


3 rd 


4 ® 


“5®“ 


7® 


8 ® 


Teacher 


91% 


94% 


96% 


91% 


93% 


83% 


91% 


Student/Self 


4% 


3% 


4% 


9% 


3% 


5% 


3% 


Peer 


2 % 


0 % 


0 % 


0 % 


2 % 


2 % 


5% 


Teacher and Student and/or Peer 


3% 


3% 


0 % 


0 % 


2 % 


11 % 


2 % 
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Appendix A 



Sample for Classroom Activities bv Grade Level 





Activities 


Teachers 


Mean Number of 
Activities 
Per Teacher 




Number 


Percentage 


Number 


Percentage 


Instruction 












Grade 2 


308 


16% 


37 


15% 


8.32 


Grade 3 


423 


22% 


49 


20% 


8.63 


Grade 4 


245 


13% 


31 


13% 


7.90 


Grade S 


328 


17% 


44 


18% 


7.45 


Grade 7 


315 


16% 


37 


15% 


8.51 


Grade 8 


321 


17% 


50 


20% 


6.42 


Assessment 












Grade 2 


225 


16% 


36 


17% 


6.25 


Grade 3 


287 


21% 


40 


19% 


7.18 


Grade 4 


162 


12% 


26 


12% 


6.23 


Grade S 


246 


18% 


38 


18% 


6.47 


Grade 7 


233 


17% 


32 


15% 


12 % 


Grade 8 


235 


17% 


43 


20% 


5.47 


Test Preparation 












Grade 3 


39 


31% 


21 


41% 


1.86 


Grade S 


28 


22% 


10 


20% 


2.80 


Grade 8 


58 


46% 


20 


39% 


2.90 


Scoring Scheme 












Grade 2 


34 


10% 


17 


12% 


2.00 


Grade 3 


75 


23% 


29 


20% 


2.59 


Grade 4 


32 


10% 


14 


10% 


2.29 


Grade S 


61 


18% 


27 


19% 


2.26 


Grade 7 


64 


19% 


23 


16% 


2.78 


Grade 8 


66 


20% 


32 


23% 


2.06 


Not Coded 












Grade 2 


24 


15% 


15 


17% 


1.60 


Grade 3 


30 


18% 


16 


18% 


1.88 


Grade 4 


15 


9% 


9 


10% 


1.67 


Grade S 


35 


21% 


17 


19% 


2.06 


Grade 7 


27 


17% 


13 


14% 


2.08 


Grade 8 


32 


20% 


20 


22% 


1.60 
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Appendix B 

Adjustment for Calculating Rate of Agreement for Classroom Activities Cndjnpr 

To obtain the correction for the total number of possible agreements, three sources of information were 

used: 

1) Within each item (e.g., #14 Problem/Item Type), die percentage of times each option was coded was 
obtained (e.g., computation, textbook-like, etc.). 

For example, 1 830 out of 3482 or 53% of the activities had die “computations” option coded for 
the problem/item type. 

2) For each item, the frequency distribution for the number of total options selected for an activity was 
obtained. 

For example, for problem/item type, 75% of the activities had only 1 option selected, 19% of the 
activities had 2 options selected, and 5% of the activities had 3 options selected. 

3) Rater’s knowledge of the coding scheme and the nature of die activities was used to determine the 
number of possible options that would need to be considered when selecting an option wi thin an 
item. 

For example, for problem/item type, if an artifact involves primarily gra phing , then a rater may 
have to consider 2 options (MSPAP-like 1 and MSPAP-like 2) when making a decision. 



To determine an appropriate number used for adjusting the total possible agreements, each item in which 
a rater could choose more than one option was considered. These items are 11, 13, 14, 15, 16, 20, and 23. 
The steps used to obtain the adjustment are as follows. Once again, Problem/Item Type is used as an 
illustration. 

1) The frequency distribution for the number of options selected per activity was examined. For the 
item, Problem/Item Type, there were some instances in which more than one option was selected for 
an artifact. While only 1 option was selected in 75% of the activities, 2 options were coded for 19% 
of the activities, and 3 options were coded for 5% of the activities. To obtain a very conservative 
estimate of agreement, it could be said that up to 2 options were considered for nearly all activities 
(except 5% of them). Therefore, since a total of 7 options are in this category, then 5 options were 
not considered in the decision-making for each activity. So the adjustment would be by 5. 

2) The information used next was the percentages of times each option was selected. For Problem/Item 
Type, 4 options were selected more than 10% of the time. They were: computation, textbook-like 
word problems, MSPAP-like 1, and MSPAP-like 2. To obtain a more liberal estimate of agreement, 
it could be said that 4 options were considered when coding each artifact in this category. There are 
a total of 7 options in this category, therefore the adjustment would be by 3 (the number of options 
that were not considered in the decision-making process). 

3) Using both pieces of information above, the options in this category were discussed qualitatively by 
the raters to determine what combinations of options needed to be considered when making various 
decisions related to coding the activities. For those activities with 2 codes, it is possible that the rater 
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had to consider 2 options for each of die codes prior to selecting each of the 2 codes. Therefore, a 
total of 4 options may have been considered for each activity. For example, one portion of the 
activity may consist of a task that could be coded as either MSPAP-like 1 or MSPAP-like 2. Another 
portion of die activity could consist of items that could be considered as either computation or 
textbook-like word problems. 

Once again, this leads to an adjustment equal to 3 (7 total options minus 4 options being considered.) 
However, in this category, the percentage of times that a total of 2 options was coded for an activity 
(19%) was considerably less than the percentage of times only 1 option was coded (75%). 
Therefore, mting this information into account with die very conservative estimate of S (described in 
step 1 above), die adjustment for the problem/item type category was determined to be 4. 



Example: 

When 2 options are coded (and only those 2 are considered) adjust by S: 

1 0 0 % 

1 1 100% very conservative estimate of 50% 



When 2 options are coded (and 4 options are considered) adjust by 3: 

0 0 100% 

1 1 100% 

0 1 0 % 

0 0 100% liberal estimate of 75% 



We select somewhere between these two estimates of 50% and 75% (adjust by 4). 



148 



152 



Appendix C 

Content Learning Outcomes for Mathematics Classroom Activities — Grade 2 





Number Of Learning Outcomes Selected 


overall 


One 


Two 


Three 


Instruction 


n=261 (85%) 


n=40 (13%) 


n=6 (2%) 




Numbers and Operations 


65% 


58% 


83% 


65% 


Geometry/ Measurement 


13% 


63% 


100% 


22% 


Statistics 


11% 


53% 


67% 


18% 


Probability 


5% 


10% 


17% 


6% 


Pattern and Relationships 


4% 


18% 


33% 


7% 


Pre- Algebra 


0% 


0% 


0% 


0% 


Algebra 


0% 


0% 


0% 


0% 


None 


1% 


0% 


0% 


1% 


Assessment 


n=167 (74%) 


n=42 (19%) 


n=16 (7%) 




Numbers and Operations 


72% 


88% 


100% 


77% 


Geometry/ Measurement 


13% 


47% 


75% 


24% 


Statistics 


10% 


33% 


50% 


17% 


Probability 


2% 


5% 


19% 


4% 


Pattern and Relationships 


2% 


21% 


56% 


10% 


Pre- Algebra 


0% 


0% 


0% 


r 0% 


Algebra 


0% 


5% 


0% 


1% 


None 


<1% 


0% 


0% 


<1% 



Content Learning Outcomes for Mathematics Classroom Activities — Grade 3 





Number Of Learning Outcomes Selected 


overall 


One 


Two 


Three 


Instruction 


n=356 (84%) 


n=52(12%) 


n=12 (3%) 




Numbers and Operations 


60% 


79% 


92% 


63% 


Geometry/ Measurement 


19% 


62% 


100% 


27% 


Statistics 


14% 


31% 


67% 


18% 


Probability 


5% 


21% 


8% 


8% 


Pattern and Relationships 


2% 


8% 


33% 


4% 


Pre-Algebra 


0% 


0% 


0% 


0% 


Algebra 


0% 


0% 


0% 


0% 


None 


<1% 


0% 


0% 


<1% 


Assessment 


n=210(73%) 


n=58 (20%) 


n=14 (5%) 




Numbers and Operations 


62% 


83% 


100% 


69% 


Geometry/ Measurement 


19% 


55% 


71% 


30% 


Statistics 


14% 


33% 


64% 


22% 


Probability 


2% 


9% 


14% 


5% 


Pattern and Relationships 


2% 


21% 


50% 


9% 


Pre-Algebra 


1% 


0% 


0% 


<1% 


Algebra 


0% 


0% 


0% 


<1% 


None 


1% 


0% 


0% 


1% 


Test Preparation 


n=23 (59%) 


n=9 (23%) 


n=7 (18%) 1 




Numbers and Operations 


57% 


56% 


71% 


59% 


Geometry/ Measurement 


26% 


67% 


100% 


49% 


Statistics 


9% 


33% 


71% 


26% 


Probability 


4% 


11% 


0% 


5% 


Pattern and Relationships 


4% 


33% 


57% 


21% 


Pie- Algebra 


0% 


0% 


0% 


0% 


Algebra 


0% 


0% 


0% 


0% 


None 


0% 


0% 


0% 


0% 



Content Learning Outcomes for Mathematics Classroom Activities — Grade 4 





Number Of Learning Outcomes Selected 


overall 


One 


Two 


Three 


Instruction 


n=201 (82%) 


n=38 (16%) 


n=2 (1%) 




Numbers and Operations 


62% 


84% 


100% 


66% 


Geometry/ Measurement 


22% 


40% 


50% 


1 26% 


Statistics 


10% 


29% 


50% 


15% 


Probability 


3% 


21% 


50% 


7% 


Pattern and Relationships 


3% 


24% 


50% 


7% 


Pre- Algebra 


0% 


0% 


0% 


<1% 


Algebra 


0% 


3% 


0% 


1% 


None 


1% 


0% 


0% 


<1% 


Assessment 


n=122(75%) 


n=32 (20%) 


n=7 (4%) 




Numbers and Operations 


71% 


91% 


86% 


76% 


Geometry/ Measurement 


21% 


63% 


71% 


32% 


Statistics 


5% 


34% 


57% 


14% 


Probability 


1% 


6% 


14% 


3% 


Pattern and Relationships 


2% 


3% 


57% 


4% 


Pre-Algebra 


0% 


0% 


0% 


0% 


Algebra 


0% 


3% 


14% 


1% 


None 


1% 


0% 


0% 


1% 



Content Learning Outcomes for Mathematics Classroom Activities — Grade 5 





Number Of Learning Outcomes Selected 


overall 


One 


Two 


Three 


Instruction 


n=257 (78%) 


n=56(17%) 


n=8 (2%) 




Numbers and Operations 


52% 


77% 


100% 


58% 


Geometry/ Measurement 


22% 


54% 


88% 


30% 


Statistics 


20% 


36% 


25% 


24% 


Probability 


4% 


18% 


63% 


9% 


Pattern and Relationships 


2% 


9% 


25% 


5% 


Pre- Algebra 


<1% 


4% 


0% 


1% 


Algebra 


0% 


4% 


0% 


1% 


None 


0% 


0% 


0% 


0% 


Assessment 


n=152 (62%) 


n=62 (25%) 


n=25 (10%) 




Numbers and Operations 


55% 


89% 


88% 


68% 


Geometry/ Measurement 


24% 


57% 


80% 


40% 


Statistics 


15% 


36% 


48% 


25% 


Probability 


3% 


5% 


40% 


9 % 


Pattern and Relationships 


2% 


8% 


28% 


8% 


Pre- Algebra 


0% 


3% 


4% 


1% 


Algebra 


0% 


3% 


12% 


2% 


None 


2% 


0% 


0% 


2% 


Test Preparation 


n=16 (57%) 


n=7 (25%) 


n=3 (1 1%) 




Numbers and Operations 


31% 


86% 


67% 


54% 


Geometry/ Measurement 


19% 


57% 


100% 


43% 


Statistics 


31% 


29% 


33% 


32% 


Probability 


13% 


29% 


67% 


21% 


Pattern and Relationships 


0% 


0% 


33% 


11% 


Pre-Algebra 


0% 


0% 


0% 


7% 


Algebra 


0% 


0% 


0% 


0% 


None 


6% 


0% 


0% 


4% 
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Content Learning Outcomes for Mathematics Classroom Activities — Grade 7 





Number Of Learning Outcomes Selected 


overall 


One 


Two 


Three 


Instruction 


n=230 (73%) 


n=75 (24%) 


n=8 (3%) 




Numbers and Operations 


45% 


79% 


88% 


54% 


Geometry/ Measurement 


26% 


29% 


63% 


28% 


Statistics 


12% 


33% 


. 50% 


18% 


Probability 


8% 


11% 


13% 


9% 


Pattern and Relationships 


1% 


3% 


13% 


2% 


Pre-Algebra 


2% 


7% 


25% 


4% 


Algebra 


6% 


39% 


50% 


16% 


None 


<1% 


0% 


0% 


<1% 


Assessment 


n=121 (52%) 


n=63 (27%) 


n=34 (15%) 




Numbers and Operations 


50% 


79% 


100% 


68% 


Geometry/ Measurement 


25% 


24% 


38% 


29% 


Statistics 


10% 


14% 


15% 


17% 


Probability 


8% 


2% 


3% 


5% 


Pattern and Relationships 


0% 


6% 


0% 


4% 


Pre- Algebra 


2% 


16% 


50% 


17% 


Algebra 


4% 


59% 


94% 


37% 


None 


2% 


0% 


0% 


1% 
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Content Learning Outcomes for Mathematics Classroom Activities — Grade 8 





Number Of Learning Outcomes Selected 


overall 


One 


Two 


Three 


Instruction 


n=245 (76%) 


n=59 (18%) 


n=13 (4%) 




Numbers and Operations 


36% 


68 % 


92% 


45% 


Geometry/ Measurement 


25% 


48% 


39% 


31% 


Statistics 


17% 1 


25% 


15% 


19% 


Probability 


3% 


12% 


8 % 


5% 


Pattern and Relationships 


2% 


7% 


15% 


4% 


Pre-Algebra 


5% 


9% 


39% 


7% 


Algebra 


11% 


32% 


92% 


19% 


None 


1% 


0% 


0% 


1% 


Assessment 


n=144 (61%) 


n=60 (26%) 


n=21 (9%) 




Numbers and Operations 


38% 


57% 


95% 


50% 


Geometry/ Measurement 


16% 


32% 


38% 


26% 


Statistics 


16% 


18% 


14% 


20% 


Probability 


4% 


10% 


5% 


6 % 


Pattern and Relationships 


0% 


2% 


0% 


2% 


Pre-Algebra 


4% 


22% 


52% 


14% 


Algebra 


20% 


60% 


95% 


40% 


None 


2% 


0 % 


0% 


1% 


Test Preparation 


n=41 (71%) 


n=12 (21%) 


n=3 (5%) 




Numbers and Operations 


29% 


50% 


100% 


40% 


Geometry/ Measurement 


24% 


50% 


0% 


31% 


Statistics 


20% 


50% 


33% 


29% 


Probability 


15% 


25% 


100% 


24% 


Pattern and Relationships 


0% 


8 % 


0% 


2% 


Pre- Algebra 


0% 


0% 


0% 


0% 


Algebra 


12% 


17% 


67% 


16% 


None 


0% 


0% 


0% 


0% 



154 



158 



Appendix D 

Response Type for Mathematics Classroom Activities — Grade 2 





Number of Response T] 


rpes Selec 


ted 


overall 


One 


Two 


Three 


Four 


Five 


Instruction 


n=124 

(40%) 


n=100 

(33%) 


n=41 

(13%) 


n=22 

(7%) 


n=14 

(5%) 


Multiple Choice/Matching 


9% 


18% 


12% 


18% 


7% 


13% 


Short Answer 


10 


25 


39 


73 


93 


29 


Numerical Value 


48 


76 


76 


96 


79 


67 


Expression 


2 


5 


20 


14 


14 


r~ 7 


Making a Chart/Table/Graph 


14 


14 


29 


55 


71 


23 


Show Your Work 


0 


11 


27 


18 


14 


10 


Short Explanation 


1 


11 


39 


36 


86 


18 


Long Explanation 


2 


6 


7 


9 


36 


7 


Creating/Posing Math Problems 


2 


1 


12 


0 


14 


4 


Figural Patterns/Diagrams 


5 


15 


7 


27 


0 


11 


Measuring 


0 


6 


12 


14 


36 


7 


Modeling/Experiments 


1 


2 


2 


0 


0 


2 


Oral 


0 


6 


7 


32 


14 


6 


Other* 


6 


4 


10 


9 


36 


8 


Assessment 


n=85 

(38%) 


n=69 

(31%) 


n=39 

(17%) 


n=22 

(10%) 


n=7 

(3%) 


overall 


Multiple Choice/Matching 


42% 


29% 


23% 


36% 


14% 


33% 


Short Answer 


2 


19 


56 


73 


71 


27 


Numerical Value 


45 


71 


82 


82 


100 


^ 65 


Expression 


1 


3 


13 


5 


57 


7 


Making a Chart/Table/Graph 


6 


19 


33 


46 


43 


20 


Show Your Work 


0 


16 


18 


32 


86 


14 


Short Explanation 


0 


15 


28 


46 


43 


16 


Long Explanation 


2 


4 


5 


5 


43 


5 


Creating/Posing Math Problems 


1 


1 


3 


5 


14 


3 


Figural Pattems/Diagrams 


0 


3 


13 


32 


14 


7 


Measuring 


0 


4 


15 


23 


14 


8 


Modeling/Experiments 


0 


1 


3 


0 


0 


1 


Oral 


0 


12 


8 


9 


0 


6 


Other* 


0 


2 


0 


10 


0 


2 
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Response Type for Mathematics Classroom Activities — Grade 3 





Number of Response Types Selected 


overall 


One 


Two 


Three 


Four 


Five 


Instruction 


n=132 

(31%) 


n=142 

(34%) 


n=72 

(17%) 


n=39 

(9%) 


n=31 

(7%) 


Multiple Choice/Matching 


9 


12 


18 


5 


16 


12 


Short Answer 


9 


28 


35 


41 


65 


28 


Numerical Value 


49 


68 


74 


80 


87 


66 


Expression 


3 


9 


8 


23 


13 


10 


Making a Chart/Table/Graph 


14 


22 


35 


46 


52 


26 


Show Your Work 


1 


15 


28 


31 


61 


18 


Short Explanation 


2 


17 


42 


69 


90 


28 


Long Explanation 


2 


4 


8 


18 


39 


9 


Creating/Posing Math Problems 


2 


4 


11 


15 


13 


7 


Figural Pattems/Diagrams 


2 


9 


15 


21 


23 


11 


Measuring 


0 


5 


13 


21 


19 


8 


Modeling/Experiments 


2 


0 


0 


13 


0 


2 


Oral 


0 


7 


11 


15 


16 


7 


Other* 


5 


1 


2 


3 


6 


3 


Assessment 


n=90 

(31%) 


n=75 

(26%) 


n=64 

(22%) 


n=25 

(9%) 


n=17 

(6%) 


overall 


Multiple Choice/Matching 


24 


28 


23 


16 


35 


27 


Short Answer 


6 


25 


33 


44 


41 


27 


Numerical Value 


51 


67 


91 


88 


94 


73 


Expression 


0 


3 


5 


28 


29 


10 


Making a Chart/Table/Graph 


8 


15 


19 


32 


53 


19 


Show Your Work 


0 


7 


33 


36 


59 


21 


Short Explanation 


6 


27 


47 


84 


82 


37 


Long Explanation 


1 


4 


9 


16 


29 


7 


Creating/Posing Math Problems 


0 


1 


0 


12 


18 


3 


Figural Pattems/Diagrams 


1 


15 


8 


16 


18 


13 


Measuring 


0 


3 


9 


8 


24 


8 


Modeling/Experiments 


0 


1 


0 


4 


6 


1 


Oral 


0 


1 


14 


4 


6 


4 


Other* 


3 


4 


9 


12 


6 


7 
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Response Type for Mathematics Classroom Activities — Grade 3 (contl 





Number of Response Tj 


pes Selected 


overall 


One 


Two 


Three 


Four 


Five 


Test Preparation 


n=3 

(8%) 


n=2 

(5%) 


n=12 

(31%) 


n=13 

(33%) 


n=5 

(13%) 


Multiple Choice/Matching 


0 


0 


8 


IS 


0 


8 


Short Answer 


0 


100 


17 


46 


80 


46 


Numerical Value 


100 


50 


50 


62 


100 


64 


Expression 


0 


0 


0 


0 


0 


3 


Making a Chart/Table/Graph 


0 


0 


75 


62 


100 


64 


Show Your Work 


0 


0 


33 


39 


60 


39 


Short Explanation 


0 


50 


67 


85 


100 


69 


Long Explanation 


0 


0 


0 


39 


40 


23 


Creating/Posing Math Problems 


0 


0 


0 


0 


0 


0 


Figural Patterns/Diagrams 


0 


0 


25 


39 


0 


31 


Measuring 


0 


0 


17 


0 


0 


5 


Modeling/Experiments 


o 


0 


0 


0 


0 


0 


Oral 


0 


0 


0 


8 


0 


8 


Other* 


0 


0 


8 


8 


20 


13 
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Response Type for Mathematics Classroom Activities — Grade 4 





Number of Response Types Selected 


overall 


One 


Two 


Three 


Four 


Five 


Instruction 


n=72 

(29%) 


n=83 

(34%) 


n=47 

(19%) 


n=25 

(10%) 


n=10 

(4%) 


Multiple Choice/Matching 


10 


13 


15 


24 


30 


15 


Short Answer 


4 


37 


34 


68 


80 


33 


Numerical Value 


68 


71 


85 


96 


100 


78 


Expression 


3 


0 


6 


16 


0 


5 


Making a Chart/Table/Graph 


8 


16 


30 


56 


50 


24 


Show Your Work 


0 


19 


21 


28 


50 


16 


Short Explanation 


0 


15 


53 


56 


80 


27 


Long Explanation 


0 


6 


6 


16 


20 


7 


Creating/Posing Math Problems 


1 


4 


6 


8 


30 


6 


Figural Pattems/Diagrams 


4 


10 


21 


4 


10 


10 


Measuring 


0 


5 


4 


12 


0 


5 


Modeling/Experiments 


0 


0 


2 


4 


10 


4 


Oral 


0 


2 


4 


12 


20 


5 


Other* 


1 


2 


10 


0 


20 


4 


Assessment 


n=57 

(35%) 


n=47 

(29%) 


n=32 

(20%) 


n=18 

(11%) 


n=8 

(5%) 


overall 


Multiple Choice/Matching 


39 


19 


56 


50 


50 


38 


Short Answer 


5 


45 


59 


83 


75 


40 


Numerical Value 


49 


77 


81 


94 


88 


70 


Expression 


0 


2 


9 


11 


25 


5 


Making a Chart/Table/Graph 


4 


11 


19 


28 


13 


12 


Show Your Work 


0 


17 


22 


56 


63 


19 


Short Explanation 


2 


11 


31 


33 


50 


16 


Long Explanation 


0 


2 


0 


11 


25 


3 


Creating/Posing Math Problems 


0 


2 


3 


6 


13 


3 


Figural Pattems/Diagrams 


0 


0 


6 


17 


50 


6 


Measuring 


0 


6 


6 


6 


38 


6 


Modeling/Experiments 


0 


2 


3 


0 


0 


1 


Oral 


0 


4 


0 


0 


13 


2 


Other* 


2 


2 


3 


6 


0 


2 
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Response Type for Mathematics Classroom Activities — Grade 5 





Number of Response Tj 


fTpes Selected 


overall 


One 


Two 


Three 


Four 


Five 


Instruction 


n=106 

(32%) 


n=90 

(27%) 


n=68 

(21%) 


n=36 

(11%) 


h=20 

(6%) 


Multiple Choice/Matching 


9 


11 


18 


17 


25 


14 


Short Answer 


7 


30 


41 


50 


65 


31 


Numerical Value 


55 


81 


90 


94 


90 


77 


Expression 


0 


4 


9 


6 


10 


5 


Making a Chart/Table/Graph 


14 


14 


34 


67 


80 


30 


Show Your Work 


0 


8 


21 


22 


60 


13 


Short Explanation 


7 


18 


38 


78 


90 


31 


Long Explanation 


2 


4 


7 


19 


40 


10 


Creating/Posing Math Problems 


0 


8 


12 


6 


5 


6 


Figural Pattems/Diagrams 


5 


9 


15 


11 


10 


10 


Measuring 


0 


7 


15 


14 


15 


8 


Modeling/Experiments 


1 


0 


2 


9 


5 


3 


Oral 


0 


0 


0 


3 


5 


1 


Other* 


1 


5 


0 


6 


0 


4 


Assessment 


n=94 

(38%) 


n=55 

(22%) 


n=47 

(19%) 


n=27 

(11%) 


n=12 

(5%) 


overall 


Multiple Choice/Matching 


50 


18 


32 


33 


17 


35 


Short Answer 


1 


40 


47 


59 


58 


32 


Numerical Value 


30 


80 


85 


96 


92 


65 


Expression 


1 


2 


9 


11 


33 


7 


Making a Chart/Table/Graph 


6 


11 


40 


33 


50 


22 


Show Your Work 


1 


13 


19 


37 


50 


15 


Short Explanation 


1 


13 


36 


56 


83 


25 


Long Explanation 


3 


6 


11 


4 


25 


8 


Creating/Posing Math Problems 


1 


6 


2 , 


7 


0 


3 


Figural Pattems/Diagrams 


1 


2 


6 


37 


42 


11 


Measuring 


0 


9 


11 


15 


25 


8 


Modeling/Experiments 


0 


2 


0 


0 


8 


2 


Oral 


0 


0 


2 


7 


17 


2 


Other* 


4 


0 


0 


4 


0 


3 
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Response Type for Mathematics Classroom Activities — Grade 5 (contl 





Number of Response Types Selected 


overall 


One 


Two 


Three 


Four 


Five 


Test Preparation 


n=7 

(25%) 


n=2 

(7%) 


n=5 

(18%) 


n= 7 
(25%) 


n=4 

(14%) 


Multiple Choice/Matching 


0 


0 


0 


14 


50 


11 


Short Answer 


0 


0 


20 


29 


50 


29 


Numerical Value 


14 


0 


80 


86 


100 


64 


Expression 


0 


0 


0 


29 


25 


14 


Making a Chart/Table/Graph 


14 


0 


40 


57 


100 


46 


Show Your Work 


14 


0 


40 


29 


50 


29 


Short Explanation 


29 


50 


80 


86 


100 


71 


Long Explanation 


29 


50 


0 


43 


25 


32 


Creating/Posing Math Problems 


0 


0 


20 


0 


0 


4 


Figural Pattems/Diagrams 


0 


0 


20 


0 


0 


7 


Measuring 


0 


0 


0 


0 


0 


4 


Modeling/Experiments 


0 


50 


0 


0 


0 


8 


Oral 


0 


0 


0 


0 


0 


0 


Other* 


0 


50 


0 


28 


0 


15 



Response Type for Mathematics Classroom Activities — Grade 7 





Number of Response Types Selected 


overall 


One 


Two 


Three 


Four 


Five 


Instruction 


n=118 

(38%) 


n=86 

(27%) 


n=56 

(18%) 


n=32 

(10%) 


n=15 

(5%) 


Multiple Choice/Matching 


7 


5 


11 


6 


13 


7 


Short Answer 


4 


31 


48 


44 


67 


29 


Numerical Value 


75 


79 


93 


91 


100 


82 


Expression 


3 


8 


18 


6 


7 


8 


Making a Chart/Table/Graph 


2 


11 


18 


53 


67 


17 


Show Your Work 


0 


14 


20 


34 


27 


14 


Short Explanation 


1 


21 


48 


66 


87 


28 


Long Explanation 


3 


6 


11 


34 


27 


11 


Creating/Posing Math Problems 


1 


6 


7 


9 


0 


5 


Figural Pattems/Diagrams 


3 


8 


0 


9 


33 


6 


Measuring 


0 


9 


14 


22 


47 


10 


Modeling/Experiments 


1 


1 


2 


12 


13 


4 


Oral 


0 


2 


4 


9 


7 


3 


Other* 


1 


0 


7 


3 


7 


3 


Assessment 


n=60 

(26%) 


n=66 

(28%) 


n=58 

(25%) 


n=31 

(13%) 


n=13 

(6%) 


overall 


Multiple Choice/Matching 


27 


8 


21 


36 


31 


22 


Short Answer 


r 2 


32 


57 


58 


85 


38 


Numerical Value 


62 


89 


91 


0 


100 


85 


Expression 


0 


21 


28 


26 


23 


19 


Making a Chart/Table/Graph 


2 


6 


17 


23 


23 


12 


Show Your Work 


0 


17 


40 


68 


46 


28 


Short Explanation 


2 


12 


21 


42 


69 


20 


Long Explanation 


7 


0 


7 


13 


54 


10 


Creating/Posing Math Problems 


0 


2 


3 


7 


8 


3 


Figural Pattems/Diagrams 


0 


2 


3 


13 


23 


5 


Measuring 


0 


5 


7 


7 


23 


5 


Modeling/Experiments 


0 


0 


0 


7 


8 


1 


Oral 


0 


2 


2 


3 


8 


2 


Other* 


0 


3 


3 


0 


0 


2 
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Response Type for Mathematics Classroom Activities — Grade 8 





Number of Response Types Selected 


overall 


One 


Two 


Three 


Four 


Five 


Instruction 


n=l 12 
(35%) 


n=85 

(27%) 


n=59 

(18%) 


n=33 

(10%) 


n=26 

( 8 %) 


Multiple Choice/Matching 


11 


4 


12 


9 


12 


9 


Short Answer 


5 


24 


42 


49 


39 


25 1 


Numerical Value 


71 


79 


85 


91 


92 


80 


Expression 


2 


13 


15 


12 


27 


11 


Making a Chart/Table/Graph 


1 


17 


24 


52 


62 


21 


Show Your Work 


0 


27 


27 


27 


50 


20 


Short Explanation 


0 


9 


51 


79 


85 


29 


Long Explanation 


5 


8 


14 


33 


46 


14 


Creating/Posing Math Problems 


0 


1 


5 


3 


0 


2 


Figural Pattems/Diagrams 


3 


7 


10 


24 


23 


10 


Measuring 


0 


4 


14 


18 


42 


10 


Modeling/Experiments 


0 


5 


2 


0 


23 


4 


Oral 


0 


2 


0 


3 


0 


1 


Other* 


4 


1 


0 


0 


0 


2 


Assessment 


n=66 

(28%) 


n=71 

(30%) 


n=44 

(19%) 


n=33 

(14%) 


n=14 

(6%) 


overall 


Multiple Choice/Matching 


33 


1 


18 


24 


43 


20 


Short Answer 


3 


18 


32 


61 


93 


29 


Numerical Value 


50 


85 


86 


97 


86 


77 


Expression 


2 


21 


23 


15 


29 


17 


Making a Chart/Table/Graph 


3 


14 


25 


39 


43 


21 


Show Your Work 


0 


38 


59 


36 


50 


33 


Short Explanation 


2 


9 


25 


52 


64 


21 


Long Explanation 


6 


10 


18 


39 


50 


18 


Creating/Posing Math Problems 


0 


1 


0 


6 


0 


1 


Figural Pattems/Diagrams 


0 


1 


9 


12 


21 


6 


Measuring 


0 


0 


5 


6 


7 


2 


Modeling/Experiments 


0 


1 


0 


3 


0 


1 


Oral 


0 


0 


0 


9 


7 


2 


Other* 


2 


0 


0 


0 


7 


1 
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Response Type for Mathematics Classroom Activities ~ Grade 8 fconO 



t 





Number of Response Types Selected 


overall 


One 


Two 


Three 


Four 


Five 


Test Preparation 


n=5 

(9%) 


n=9 

(16%) 


n=15 

(26%) 


n=16 

(28%) 


n=6 

(10%) 


Multiple Choice/Matching 


0 


0 


0 


6 


0 


2 


Short Answer 


0 


10 


20 


38 


100 


38 


Numerical Value 


SO 


50 


87 


100 


83 


85 


Expression 


0 


0 


20 


13 


0 


9 


Making a Chart/Table/Graph 


20 


30 


33 


50 


83 


48 


Show Your Work 


0 


30 


47 


31 


17 


29 


Short Explanation 


0 


0 


60 


88 


83 


59 


Long Explanation 


0 


50 


27 


25 


100 


41 


Creating/Posing Math Problems 


0 


0 


0 


0 


0 


3 


Figural Pattems/Diagrams 


0 


10 


0 


31 


17 


14 


Measuring 


0 


0 


7 


13 


0 


10 


Modeling/Experiments 


0 


20 


0 


0 


0 


8 


Oral 


0 


0 


0 


6 


0 


5 


Other* 


0 


0 


0 


0 


17 


3 
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Appendix £ 

s 

MSPAP-like Levels for Mathematics Classroom Activities -- Grade 2 





Number of Task Types Selected 


overall 


One 


Two 


Three 


Instruction 


n=276 

(90%) 


n=31 

(10%) 


n=l 

(0%) 


Computation/Equation 


48 


74 


100 


51 


Traditional Word Problems 


5 


19 


0 


6 


Graph, Table, Model, Pattern, etc. 


14 


45 


100 


17 


MSPAP-like 1 


24 


58 


100 


28 


MSPAP-like 2 


5 


3 


0 


5 


MSPAP-like3 


3 


0 


0 


3 


Assessment 


n=132 

(59%) 


n=71 

(32%) 


n=18 

(8%) 


overall 


Computation/Equation 


46 


89 


94 


64 


Traditional Word Problems 


5 


70 


100 


34 


Graph, Table, Model, Pattern, etc. 


12 


18 


28 


16 


MSPAP-like 1 


29 


20 


78 


31 


MSPAP-like 2 


6 


3 


0 


4 


MSPAP-like 3 


2 


0 


0 


1 
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MSPAP-like Levels for Mathematics Classroom Activities — Grade 3 





Number of Task Types Selected 


overall 


One 


Two 


Three 


Instruction 


n=364 

(86%) 


n=49 

(12%) 


n=9 

(2%) 


Computation/Equation 


38 


78 


89 


44 


Traditional Word Problems 


4 


53 


89 


12 


Graph, Table, Model, Pattern, etc. 


11 


14 


11 


12 


MSPAP-like 1 


32 


47 


100 


35 


MSPAP-like 2 


8 


8 


11 


8 


MSPAP-like 3 


7 


0 


0 


6 


Assessment 


n=196 

(68%) 


n=66 

(23%) 


n=18 

(6%) 


overall 


Computation/Equation 


38 


82 


100 


54 


Traditional Word Problems 


3 


67 


100 


26 


Graph, Table, Model, Pattern, etc. 


7 


8 


17 


10 


MSPAP-like 1 


34 


39 


83 


40 


MSPAP-like 2 


11 


5 


0 


8 


MSPAP-like 3 


7 


0 


0 


5 


Test Preparation (gr. 3, 5, 8) 


n=38 

(97%) 


n=0 

(0%) 


n=0 

(0%) 


overall 


Computation/Equation 


8 


0 


0 


10 


Traditional Word Problems 


0 


0 


0 


3 


Graph, Table, Model, Pattern, etc. 


3 


0 


0 


5 


MSPAP-like 1 


32 


0 


0 


33 


MSPAP-like 2 


26 


0 


0 


26 


MSPAP-like 3 


32 


0 


0 


31 
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MSPAP-like Levels fear Mathematics Classroom Activities — Grade 4 



1 

i 





Number of Task Types Selected 


overall 


One 


Two 


Three 


Instruction 


n=188 

(77%) 


n=52 

(21%) 


n=5 

(2%) 


Computation/Equation 


40 


83 


100 


50 


Traditional Word Problems 


7 


46 


60 


17 


Graph, Table, Model, Pattern, etc. 


8 


19 


40 


11 


MSPAP-like 1 


35 


40 


100 


38 


MSPAP-like 2 


8 


10 


0 


8 


MSPAP-like 3 


1 


2 


0 


1 


Assessment 


n=93 

(57%) 


n=47 

(29%) 


n=21 

(13%) 


overall 


Computation/Equation 


55 


96 


95 


72 


Traditional Word Problems 


9 


70 


91 


38 


Graph, Table, Model, Pattern, etc. 


7 


13 


29 


12 


MSPAP-like 1 


26 


17 


81 


31 


MSPAP-like 2 


1 


4 


5 


3 


MSPAP-like 3 


2 


0 


0 


1 
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MSPAP-like Levels for Mathematics Classroom Activities — Grade 5 





Number of Task Types Selected 


overall 


One 


Two 


Three 


Instruction 


n=259 

(79%) 


n=52 

(16%) 


n=15 

(5%) 


Computation/Equation 


31 


73 


100 


41 


Traditional Word Problems 


6 


54 


73 


17 


Graph, Table, Model, Pattern, etc. 


13 


14 


40 


15 


MSPAP-like 1 


33 


54 


80 


38 


MSPAP-like 2 


9 


6 


0 


8 


MSPAP-like 3 


9 


0 


7 


7 


Assessment 


n=149 

(61%) 


n=60 

(24%) 


n=33 

(13%) 


overall 


Computation/Equation 


40 


95 


100 


62 


Traditional Word Problems 


3 


75 


97 


35 


Graph, Table, Model, Pattern, etc. 


8 


10 


21 


12 


MSPAP-like 1 


31 


18 


79 


35 


MSPAP-like 2 


8 


2 


3 


6 


MSPAP-like 3 


11 


0 


0 


7 


Test Preparation (gr. 3, 5, 8) 


n=25 

(89%) 


n=2 

(7%) 


n=l 

(4%) 


overall 


Computation/Equation 


4 


100 


0 


11 


Traditional Word Problems 


0 


0 


100 


4 


Graph, Table, Model, Pattern, etc. 


4 


0 


1 


4 


MSPAP-like 1 


52 


100 


100 


58 


MSPAP-like 2 


8 


0 


100 


11 


MSPAP-like 3 


32 


0 


0 


29 
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MSPAP-like Levels for Mathematics Classroom Activities — Grade 7 





Number of Task Types Selected 


overall 


One 


Two 


Three 


Instruction 


n=268 

(85%) 


n=43 

(14%) 


n=4 

(1%) 


Computation/Equation 


38 


84 


100 


45 


Traditional Word Problems 


8 


58 


75 


15 


Graph, Table, Model, Pattern, etc. 


4 


21 


25 


6 


MSPAP-like 1 


35 


35 


100 


36 


MSPAP-like 2 


13 


2 


0 


11 


MSPAP-like 3 


3 


0 


0 


2 


Assessment 


n=126 

(54%) 


n=77 

(33%) 


n=24 

(10%) 


overall 


Computation/Equation 


64 


96 


100 


79 


Traditional Word Problems 


5 


58 


88 


34 


Graph, Table, Model, Pattern, etc. 


2 


9 


21 


9 


MSPAP-like 1 


19 


34 


92 


34 


MSPAP-like 2 


7 


3 


0 


5 


MSPAP-like 3 


4 


0 


0 


2 
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MSPAP-like Levels for Mathematics Cla^croom Activities — firarfa ft 








Number of Task Types Selected 


overall 


One 


Two 


Three 


Instruction 


n=262 

(82%) 


n=51 

(16%) 


n=7 

(2%) 


Computation/Equation 


38 


80 


100 


46 


Traditional Word Problems 


8 


55 


71 


17 


Graph, Table, Model, Pattern, etc. 


6 


14 


57 


9 


MSPAP-like 1 


26 


39 


71 


30 


MSPAP-like 2 


13 


12 


0 


13 


MSPAP-like 3 


8 


0 


0 


7 


Assessment 


n=160 

(68%) 


n=57 

(24%) 


n=15 

(6%) 


overall 


Computation/Equation 


56 


90 


100 


67 


Traditional Word Problems 


3 


61 


73 


23 


Graph, Table, Model, Pattern, etc. 


3 


21 


33 


11 


MSPAP-like 1 


18 


25 


87 


24 


MSPAP-like 2 


14 


4 


7 


11 


MSPAP-like 3 


6 


0 


0 


4 


Test Preparation (gr. 3, 5, 8) 


n=52 

(90%) 


n=5 

(9%) 


n=l 

(2%) 


overall 


Computation/Equation 


12 


100 


100 


21 


Traditional Word Problems 


0 


20 


0 


2 


Graph, Table, Model, Pattern, etc. 


2 


0 


100 


3 


MSPAP-like 1 


27 


80 


100 


33 


MSPAP-like 2 


27 


0 


0 


24 


MSPAP-like 3 


33 


0 


0 


29 
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